How to turn down someone's invitation to lunch?

Someone I know asked me to come to their house for a lunch with family and friends. I really don't want to go because I know it'll be anxiety-inducing and I'll feel uncomfortable. I tried to use some excuses like 'I can't leave my pets alone' & 'I don't wanna bother you'. But they were insistent and I couldn't come up with any other excuses so I just said okay. What do I do? They might be mad if I go back on my word. Would they find it suspicious if I say I got sick? How can I reject their invitation without coming off as too rude?