AITA for not wanting my sisters "homeless" boyfriend move in with us?

I (24F) live with my mother and my sister Amy (20F) in a three bedroom appartment. Amy's work BFF Simon (32M) was married, and his wife was very suspicious of their relationship. This created friction and made him take breaks from seeing Amy when they no longer worked together. Through shared friends they met again and he told her he was leaving his wife and wanted to be with her. They started dating but couldn't do it openly since he had not told his wife about it. When his ex-wife found out she wouldn't let him sleep at their appartment and so he slept over at our place. Soon he was starting to "drop by", staying so late he might as well sleep over.. and then stay one night more. Turning up uninvited on Christmas day and staying 3h etc.

I'm a pretty anxious person, and this drove me mad! I was pretty much just lucky they didn't walk in when I was walking around naked while getting dressed, or on the toilet with the door open, because there would be no warning. We also have a strict "no conflicts in front of guests" style which made me even more of a pressure cooker. I couldn't even sit in the living room without hearing them...you know.

All I wanted was to be asked if he could come over and know 24h in advance if he was sleeping over. I was willing to have a discussion about it but was told I was being unreasonable, controlling and heartless. My mom said I was being unempathetic not giving a place to stay for someone who was "more or less homeless" (his ex did let him back into the app. btw but understandebly he was pretty uncomfortable being there). To my requests she just said "we don't live in a catholic school for nuns" and that I was in effect breaking them up if we were to live by my suggestion. She is very quick to consider partners as family, inviting Amy's previous bf to vacation with us after they had been dating for 2 weeks (he did kind of know him before though, he was one of my old classmates) and said I was ostracizing him when I didn't invite him to my birthday lunch that was close family only. Am I being too extreme?