How to be kind when you want to scream no?

I picked my son up from his mother's. My son told me here mother wanted to come out and talk very nice hugs.then can we go to dinner with them and my brother in law then divorced 6 years. Her daughter cheated after i.v.f treatment. I lost everything. I just said set a date. They were family for 20 years. But I wish I just said no.they are getting old .I let them live with us. I paid it all.i want to.and I dont?