Why wouldn't you let someone through?

I'm so confused by people sometimes. I kind of just want to find out if I'm insane, or if I have it right.

Times to let someone through:

• there's two more people in your group than in theirs, regardless of the pace of group in front of you (exception: the whole course is jam packed and you're waiting every tee/green, the group in front of you is waiting every time, etc...then you're all just screwed and need to relax)
• there's an entire hole open in front of you and a group of any size catches you at any point before the 18th tee
• the person catching you is a single at any point

I just wonder because we had a fairly open course on a Monday morning, just me and one friend (both 14 handicaps if that matters), and we caught a group on the 5th tee after having to wait for them on the 3rd and 4th tees and approach shots. They literally sped off without saying anything as they finished their tee shots from the whites and could se us standing on the blues already. We were confused, but focused on them for that hole and the next and noticed it looked like one outstanding golfer, one pretty good, and two bad. The two better guys were scrambling each paired with a worse guy. I swear they were doing little mini-lessons. Overall, they were on a fine pace (I bet like 3:45-4:00), but the course was empty enough you couldn't see anyone on any hole in sight in front of them.

We got to the next hole and once again are politely waiting back at a distance, they make eye contact, and say nothing. So my buddy says "Hey, would y'all mind if we play through?" You can tell three of them are like who cares, it's two guys walking they seem good sure. The best golfer, however, has the audacity to ask back "Really? Are we holding you up?"

Now, we politely handled it with a "Yes, thank you man, appreciate it!" And proceeded to hit two of the best tee shots of the day (me a 235 yard center fairway driver, buddy a 240 yard center fairway 3-wood) with the least amount of time I've ever been in a tee box and the most pressure I've felt in quite some time LOL.

We hit our approaches, got to the green, and the guy who was clearly excellent (I would bet scratch or very near to it, maybe even a +1) had driven the green. Short par 4 (probably 330 from the tees they played, and it was 380 for us - biggest gap between tee boxes on the course).

Did being really good and knowing they weren't "playing slow" blind this guy to good etiquette? Were my buddy and I supposed to lose 5-10 minutes a hole the remaining 12 holes and end up at a round a full hour+ longer than it needed to be?