connection problems but just to one server

I have played on two servers. Only started the game a week ago, but I've been obsessed.

The first server I played on, is a server a friend rented from one of those providers. I won't say the name, but it seems to have been working great, although when the game has patched it seems like the guy has to manually patch it? I am not sure.

The other server, is one another friend is running on his own hardware. Connectivity to that server has been very unreliable. There are maybe 8 other people playing on it at times, seemingly without issues. I've been playing more though, so I am mostly by myself.

By default, I use a paid VPN (I won't say the name, this is not an ad, but it is an extremely well known VPN provider.) and when I first tried to connect to it, it would not work.

The first thing I tried is to whitelist that server's IP, but it kept happening.

I did try turning the VPN off entirely, and I was able to play, but I keep having the hardest time connecting. It is really frustrating actually.

Naturally the internet is involved and that makes this sort of thing hard to troubleshoot, but are there any IP's that I would also need to exclude?