How much does grad school prestige matter?

Hi all, I'm halfway through my undergrad studies in physics, and my goal is to pursue theoretical physics in grad school in the realm of gravity and relativity. I understand how brutally competitive it is to get into good grad schools for this type of thing, and the reality is that I'm not one of the best students in my university's physics program. I'm quite ahead in my coursework--my third year is starting and I'm going to be taking graduate courses in QM and GR. However, my physics GPA is ~3.7, and so far I don't have much research experience to speak of. I have a sort of mentorship going with a theorist at my university, but he is very very busy and we haven't been able to do much since it started a few months ago. (Any tips for getting into some level of theoretical physics at the undergrad level would be insanely helpful--every time I ask an adviser or professors or anyone about this it's very discouraging!!)

So even if I really improve my application in the next year, I know I have a slim chance at getting into a very 'prestigious' grad school. This field is so competitive, I have to wonder, would my career as a theoretical physicist essentially be snuffed out if I don't go to a highly ranked grad school? How important is this really?