Joining parties on series x/s

Me and my buddy both have the series s and both have mw3. We have gotten into the same lobbies before but can’t join each others parties. Is this just not possible or is there a fix? Thanks!