My nonmonagamous partner wants us and his other partner to live together

Tl;dr: my partner wants himself, his other partner, and myself to live together and says that it's a complicated issue that he wants to work on.

Please bear with me, I don't use Reddit much and this is weighing heavily on me.

My partner (26M) and I (24FTM) have been together for a little over a year. We actually met on Reddit and have hit it off wonderfully. He's a wonderful man and he makes me very happy. Right from the start, he was honest with me that he is ethically nonmonogamous. I told him that I had no issues with people that prefer that relationship style but I am monogamous so we could be friends. Well, he knocked me on my @ss (figuratively) and I fell for him quite hard. We get along very well and adore eachother.

The goal has been to eventually close the distance between us. He lives almost 1000 miles away (or 1600 km) and the plan was that I would move to him when my occupation would allow me to transfer. Due to multiple personal matters as well as professional matters, that plan shifted to he would move here once my income becomes more steady. Out of the both of us, I am the only one working or bringing in income as he lost his job not long after we started dating. I was and am fine with being the breadwinner in our relationship. The only thing preventing him from moving here to be with me is my hours constantly shift so I can't know for sure that I will always have the money to afford a home for us due to the cost of living in my area.

When we started dating, he was already seeing someone as well as myself and over time he's developed multiple other relationships, all of which have ended except for his most recent one. They've been together for about a week and things are going well so we have been talking about our future and what it will look like. I have always taken it incredibly hard when he starts seeing someone new and it tanks my mental health for a short period of time. Once I get over that feeling, I am fine again and don't generally hold any negative feelings about his relationships. But I have always said that while I will be friendly towards his other partners, I have no desire to spend time with them and will more than likely just leave them be while they spend time together. I've mentioned that being around them being intimate would make me uncomfortable and I would not like to share spaces with them. He claimed that he understood and that I wouldn't have to if I didn't want to. I believed him as he's always been a considerate man and has treated me very well.

Cut to last night and early this morning. He started talking about him, his other partner, and myself spending time together and doing things such as cooking meals together. I told him I didn't want to do that and that I would just avoid the both of them when they were together because I don't want to spend time with his other partners. I made a joke that I would basically be a ghost in my own home since I would be there but out of sight. He insists that he's okay with me wanting to remain distant from his other partner and that he would always make time for me. Throughout the conversation, I got the feeling that he was trying to say we would all live together despite conversations earlier in our relationship in which I said I wouldn't be comfortable with that and him acknowledging that and agreeing not to cohabitate with other people. I reminded him of this and brought up how badly I take it when he starts new relationships, mentioning that the feeling is getting worse as our relationship progresses.

He proceeded to ask if I didn't want to live with other people at all and I told him that I didn't. He tried to insinuate that it could be because I was scared someone would "steal" him away from me and that being together in person would help provide some security for me. I told him that wasn't it and that the thought of seeing him with someone else makes me physically ill. He said that made things complicated, that it was something he would want to work on because it was important to him and that he didn't want to have to choose between partners. I said that if he wanted to live with his other partner then I would understand as it would provide less obstacles for his other relationships since living with me would limit what could be done with others. He said he didn't want that and continued to attempt to get me to come around to the idea because he doesn't want us to be long distance forever. I told him that what he wanted would seriously affect our relationship and then asked him if he would really be okay with asking his other partner to live with someone that would avoid them at all costs but he didn't have an answer for that. "It's something that we'll just have to see about".

What should I do? Should I try to come around to the idea and push my comfort aside to prevent damage to my relationship? Is there any way that I can explain to him that I will never want to share a space with his other partners? I genuinely don't know what to do. I can see why he wants everyone to be together but I would become seriously mentally unwell and it would completely irradiate certain aspects of our relationship.

If you've made it this far, thank you. I'm sorry for the rambling but I have no one I can turn to for this and it's eating me alive.