I think I have a wasp nest in my walls, do they die off in the winter?

Last year, I noticed that wasps had been building a nest inside a soffit outside my house. I should've done something thing, but it was near the fall and I thought the nest would just die off. Come spring this year, I did not see any movement in that soffit, so I thought that's it.

However, this summer, I started noticing wasp in the house. Most are dead, some will fly around confused before we kill it. We noticed about one new wasp a day in the house during the summers. Now that's snowing, I think whatever wasps in the house must be dead as we haven't seen any live wasps in the house for a few weeks.

I had tried to observe the house during the summer and autumn near sundown to see if I could see any wasps entering the house, but couldn't find any. I did not realize I was allergic to stings until this summer when I got a pretty good welt that went up my entire arm. My wife was cleaning up mantle a few weeks ago, saw dead wasp, and for some reason used her bare hands to clean them. She got a nice set of hives, so I am assuming she's allergic too. We also have a toddler, who I am sure is allergic (we do have epipens just in case), thankfully he knows to point them out when he sees them and doesn't go near them.

I do have a friend who is a bee keeper, but he said there's not much you can do unless you see the actual area they are entering the house.

What steps can I do to prepare for next summer so I don't have an infestation in my house next year? Should I be worried about the nest that might be in my house now?