Did I make the right call?

Was out for day 3 of Ohio gun season, with a friend. We've had some solid luck so far. My friend got a really nice doe on opening day. Saw nothing yesterday, which had me a little bummed out but kind of expected since we were hunting the same spot that we took the doe on. That leads us to today, Wednesday. Had a really young doe come in at about 4:15-ish. Didn't have spots, but it was walking the same path the other doe took, and we could tell it was almost half the size of the first. I decided that if it was still there by 5, then I would take the shot. We spend the next 40 minutes watching it, and by the time 5 rolls around my friend has me convinced to pass on it. Figure, something else would walk in between 5 and close of day. Surprisingly, a doe walked in on our right side, and it was followed by a buck, but I couldn't take the shot because there were houses behind both deer (about 150-200 yds). We hoped they would move a little more to the right for a good shot, but the doe huffed and they both booked it out of there.

I'm torn. On one hand, I feel that sparing the young one was the right call because it gives it the chance to make it to next year and get a little bigger. On the other hand, it would have put an end to what has been a somewhat disappointing season for myself. I spent a good chunk of time this year for bow season and came up empty handed due to some setbacks with equipment. I made sure that I was set for gun season, and things are looking good for tomorrow since the deer are coming back to our spot already, but I can't help but feel that not taking the shot today was a mistake.

What would you have done in my shoes?