Lesley-Anne Down says ‘everybody thought’ Patrick Swayze was having an affair with her, but he was really ‘out all night’ with this costar