My boyfriend blatantly ignores me in front of his ex, is this a red flag?

My boyfriend [20/M] and I [21/F] have been dating for six months now, and we both got out of long term high-school relationships semi recently (his four year relationship [20/F] ended over a year ago and my five year [23/M] ended around 10 months ago). Both us and our exes go to the same college and we're all art majors so we're bound to run into each other every now and then. I get a little anxious when I'm with him and I spot my ex; since I'm the one who initiated the break up, I can't help but feel like I'm only hurting him further when he sees me happy with "some other guy." However, I won't pull my hand away or seem shy of my current boyfriend. In my heart, he is the most important thing to me so it makes sense to be confident in us rather than be afraid of hurting a guy that I no longer have ties with.

But my partner doesn't seem to share the same mentality. Whenever we hold hands in public, he'll keep an eye out for his ex's car to make sure that she's not in the same vicinity. If he sees her down the hallway, he'll snatch his hand away and mutter her name to me under his breath as some kind of excuse, and then ignore me until we are very, very well out of sight.

One time I was starving and having a really crappy anxiety day, and he asked me out to one of the food eateries around campus saying that he'd pay and everything. I cannot begin to explain how badly I didn't want to leave my bed or see anyone or have to socialize, but he seemed like he really wanted to go so who was I to say no to him? I got kind of excited too once I saw the building in sight and had worked up quite an appetite for the chicken they sell. Once we were about 80 meters away, he froze and asked if we could go somewhere else. Why? Because he saw his girlfriends car in the parking lot, and while she might not have been in the restaurant we were going to (there's a Starbucks, a popular pizza joint, dorms, and class halls literally surrounding this parking lot), he didn't want to risk it.

So my question is, Reddit, is his behavior normal? Is he not over his ex like he says he is? She broke up with him and he will confide his fear, anger, and hurt about her to me (which I'm actually more than fine with because everyone needs someone to talk to about these kinds of pain and I will always accept him) but are all of these red flags?

TL;DR My boyfriend is hyper aware of his ex and ignores me when she's around, should I be concerned? What should I do about this?

NOTE: I have talked to him about this before and he said that he just gets nervous, but the way he overreacts when he sees her makes me feel like there's more.