Oh yeah, that's definitely it. Im just surprised popular culture usually takes that into account when creating worlds with giants. 

Perhaps because being at that size they would have to consume larger amounts of food therefore having a smaller population especially if they were on our earth with everything the same size except them 

I thought it was interesting because whenever I remember books / stories with giants, their worlds always see to have very few giants. 

If we lived in a world where humans were 200 ft giants, the world would have a much smaller population.