Dealing with (my) toxic boomer grandparent

I will try to keep this short and sweet. I really just need to rant but i'll take advice if you have any. I am a young mom (24 with 2 under 2) and we live on my grandparents' property. They agreed to let us stay because the mobile home on their acreage is vacant. My grandad has always been a total grouch, and never really spent time with us growing up. All he does is lecture us and brag about how great he is. He is a narcissist. In fact I found out when we were young he told my grandma she had to get HIS permission to see us because he wanted his own free time (which is like what?? we only saw them once a month max but ok..) Anyhow, fast forward to now - he insults DH, and compares him to my father (whom I cut off for certain reasons), tells us how privileged we are to live in a society with a high minimum wage (even tho we cant even afford to live and are barely scraping by), claims he "raised" me and my siblings (which isnt true, we barely spent time with him specifically) etc...theres so many things. I think he is going senile. My toddler is scared of him. Is it reasonable to just stop talking to him altogether?