No vibes at all with spouses family (33F) (32M)

We have been married 1 year. Our families never really like each other and there was no vibes. We as a couple were fine but we knew deep down our families didn’t mesh well and come from very different socioeconomic statues as well. I don’t like her brother or SIL at all and I think the same for them. There isn’t any vibe. When we visit them we just sit in the couch, no ones speaks and it’s always quiet. Or they make indirect remarks about certain things that I don’t like. It’s just not fun.

I am wondering what to do? Limit contact or what?