AITA for wanting my family time to be uninterrupted?

Am I wrong for being upset that my boyfriend had a 5 minute conversation on the middle of me, our baby and him having lunch together? He keeps calling me a jealous bitch.