AITA for skipping my friend’s birthday dinner after he invited my ex

My friend Alex had his birthday dinner at a restaurant last month and he invited both me and my ex, Sarah, even though he knows we broke up pretty recently. I told him a couple days before that I wouldn’t be comfortable sitting across from her all evening, so I stayed home. He texted me the next morning saying I made the night awkward and that I should’ve just shown up because it was his birthday. I get that it was his celebration, but I also didn’t want to sit through an uncomfortable meal. We haven’t really talked since then, and a couple of our other friends think I overreacted by not going at all. I’m wondering if I should apologize just to smooth things over, but I’m also not sure I actually did anything wrong by setting that boundary.