At what age would you un-gate your kids bedroom door if you lived in a 2 story house?

So, my daughter is 3.5. She is perfectly capable of going up and down the stairs by herself and does so multiple times a day. Our ground floor (a open-concept layout that consists of the living room, dining room and kitchen all in one) has also proved to be fairly kid safe. But I have not yet let go of the crutch of keeping a baby gate on her bedroom door because I do have a fear of her falling down the stairs if she gets up and tries to go downstairs on her own at night (it doesn't help that if the stair light is off she has to stretch precariously to hit the lightswitch).

I know she's probably big enough to have the gate off her door but giving her nighttime access to the stairs freaks me out. We have some baby gates in storage that would could theoretically put at the top of the stairs (so at least she has access to the whole top floor, I actually wouldn't mind her being able to get to my room in the night if she needed me) but she's past the weight limit for those to be safe (plus, the top of our stairs is weird as on one side we'd have to screw the gate into what seems to be a metal stud, and our drill wasn't successfully able to drill into that).

So, what do other parents of kids in 2 story houses do? At what age do you remove all the gates and let them just exist with a set of stairs and assume they won't fall down and bust their head open in the middle of the night? Is this a rational fear or an irrational one?