AITA for not wanting to visit my grandfather who's health is declining?

For background I 30F grew up living with my grandmother. I am estranged from my mother's side of the family. My dad (57m) and I are close but I have always been closer with my grandma. My dad was always around growing up and is a good father. However, he tends to pressure and guilt trip me at times to do things I do not want to do. Nothing sinister, things like spend time with my grandfather, get upset if I didn't call him enough. More or less typical parent things, but he could have gone about things much better. My dad would often take the guilt trips to an extreme level and become very passive aggressive. I am a very anxious and timid person. I can't handle disappointing him.

Getting to my grandfather. My grandma divorced him when I was young because he was abusive. I witnessed some of this abuse growing up. My grandma has since told me that the abuse happened all the way back to when my dad and uncle were kids. I have also heard my dad and uncle joke about the abuse they grew up with.

After the divorce my dad would still take me to see my grandfather despite the fact I was uncomfortable. I do not in any way fault him for doing this as I was a child and he wanted us to have a relationship. However I was always uncomfortable. I had seen my grandfather hold a gun to my grandma's head as a child. I didn't tell my dad this until I was an adult. I finally did when I got engaged and told my dad I didn't want my grandfather to come to the engagement party for my wife and I. I listed this reason and other things like the creepy messages my grandfather sent me in college. Religious coded messages about how I needed to submit myself into a husband (I'm gay). My grandfather is very racist, homophobic, misogynistic, narcissistic etc. My dad understood and was fine with me not wanting my grandfather there.

Now we're back today. My grandfather is in poor health and I haven't spoken to him in years. He blocked me years ago on Facebook for calling him out for being racist. We also live in different cities. My dad has been asking me to go and spend time with my grandfather as he has cancer and isn't doing well. I hate that he isn't doing well because no one should have to go through cancer. However, I do not want to see this horrible man.

My wife tells me I'm not wrong and I should just stand up to my dad about it. I'd feel like a huge asshole if I said no. Mostly because I don't want my dad to be disappointed in me and honestly I'm afraid of the guilt trip or other fallout that I'll receive. Would I be the asshole if I told my Dad I won't be going to see my grandfather or should I just suck it up and visit once? How can I approach this conversation with my dad?