New to BJJ... Afraid to put my weight into people?

Im 6' 183lbs and have been rolling with much bigger guys who have no problem putting every ounce of weight into their shoulders, against my chin. When the roles are reversed, I find myself supporting myself with my arms and legs, as if Im going to hurt somebody by using my weight against them. This, of course, leads to losing my position. I think this comes from growing up with little brothers and having daughters... You wrestle and play, but remain reserved as to avoid really hurting them. Is this something that will pass with experience? Should I say f@#k it and get over the fear that Im hurting someone(even though, logically, I am not)?