How much does residency location matter for Fellowship?

I'm a MS4 who is finalizing my rank list. I am struggling to choose between a more prestigious state program or my own home state program, and the question is, will having a slightly more prestigious program make a huge difference for fellowship?