Multiplayer with more that 2 people problems.

I played through the game with 1 other person online with minimal issues, but as soon I tried to invite 3 friends to play with us, we started having a stupid amount of problems getting games going. We are all starting new characters, have up to date versions, and have added each other as friends, yet every time we try to play someone can't see the server. The inability to see the server rotates so it's not just one person. Once all 4 of us are in the server within a few minutes our game splits up and some of us can't see the others, or someone will have a ridiculous load time then crash and be unable to see the server again. We all have decent internet and can play other games online together so that shouldn't be the problem either. Has anyone else had success with actual multi-player and not just a co-op? Are there any known fixes to our problems and are others experiencing them? I know some are at least having problems with seeing servers. I really enjoyed the game and wanted to play with my other friends, but unless these gamebreaking problems are solved i guess were all quitting.