AITAH for kicking my boyfriend out right after he arrived?

My (25F) boyfriend (25M), let’s call him Adam, and I have been together for about a month and a half. Important to note that we are coworkers. I’ve had issues with him ignoring my boundaries by looking at my phone and not respecting when I give no as an answer.

Last night we were on the phone and I was telling Adam about a male coworker who makes me uncomfortable and how I was glad he didn’t come swimming with us so he didn’t see me in a bathing suit. For the record, many of my female coworkers also feel weird around him and have we have all spoken about it together.

Adam said that he’s a really nice guy and shouldn’t make me feel uncomfortable then proceeded to begin making jokes, saying things like “free him.” He also said that if him swimming with me made me uncomfortable then I shouldn’t go swimming (which is part of my job).

All of the comments he made led to me feeling like he wasn’t taking me seriously and I realized that he was laughing while I was upset so I told him not to come over for the night he texted me saying he was still coming over and asked if it was okay I said “fine, but we’re talking about this.”

Adam arrived about an hour later and came up to my room where he immediately looked at my phone that was open to texts. I was folding laundry at the time so I whacked him with a t-shirt and told him to stop.

He then started baby talking my dog and saying “we’re not mad, but she’s mad” which also made me feel like he wasn’t taking things seriously. I started to talk about why I was upset and he was brushing it off and laughing so I told him to get out. He didn’t listen so I raised my voice until I was screaming for him to get out. My roommate came out and told him he had to leave and we threatened to call the cops if he didn’t. He finally left after that and texted me saying he should call the cops after I assaulted him with a t-shirt.

My question is, did I overreact and should I have heard him out? I haven’t heard from him nor texted him since, but I assume he’s expecting an apology and I’m not ready to give one.