AITA for going back on my promise for cat-sitting

So my uncle and his wife are going on holiday, and previous years I have agreed to help them watch their cat while they were gone. Once I fully stayed at their place, the second time they left me their car and I could drive to and from their place to feed it and stay a little. In a car the drive is 1,5 hours, and with public transportation it is 3,5 hours.

So now they are going on a holiday again, and I can't stay at their place since I have work too, and it just wouldn't work out with public transport times and my work times.

So they suggested they could drive the car to my place.

I also have 3 cats and live in a two room apartment (one bedroom and one livingroom) so at first I was apprehensive, but they talked me into it, so I agreed.

But thinking about it a bit more I thought maybe it wouldn't be a good idea, since 1. I have my own cats and four cats wouldn't work in my apartment. 2. Since it's a small apartment I can't really separate the cats, my own cats paw at the bedroom door if i it and are distressed. 3. I have a lot of work the two weeks they are gone so I couldn't keep an eye on them if a fight broke out. 4. Their cat is an outside cat, while mine are inside cats, I live on the second floor so she couldn't go out, and I don't know what bacteria and viruses she might have to bring over to my cats.

I felt bad just refusing, so I researched a little and found a cat shelter that can watch the cat for half the price of other places where pets can be watched.

I called them and explained my worries about having their cat over, and told them about the shelter option. They thought my worries were illogical, and got offended that I suggested their cat could make mine sick, and that I'm just being dramatic. Then they complained the price for the shelter was too much.

To compare last year when I had their car they paid for gas for the week I watched the cat, and for two weeks of stay at the shelter would be approximately the same price, so it's not that much. They told me I can't be relied on, and that I'm acting like a relative of ours that's a little crazy and always creates drama for no reason and does what fits her the most. I got upset and told them that if my cats were to get sick because of their cat, the vet bill would be ten times the shelter option.

We have not spoken since, and I'm not sure what they are choosing to do with their cat. So am I the asshole for saying no, when initially I agreed.

Between the time I agreed and then called them again a week had passed and it was still 3 weeks before they had to go on holiday so not the last moment.

Should I have just tried to see how the cats reacted and not been overthinking the risks?