Should I have my fb over while child is sleep?

I am a 22f and have a 1 year old. I have full custody. I've been meeting a guy for sex for a little over 2 years. We usually go to his apartment but his sister needed to come stay with him so his house isn't really an option anymore. We've been trying to work out some kind of schedule but we have conflicting work schedules. Would it be a bad decision to invite him over after I put the baby to sleep? The baby sleeps through the night in their own room. We've been trying out doing it in the car and other locations around his complex but I'm really uncomfortable doing that. Also me and the guy aren't together nor are we planning to be, it's just sex if that makes a difference.