Partner keeps hinting at using drugs with his friend when he visits him.

My partner has a friend who recently lost his dad due to cancer, his friend has been coping by using ketamine. My boyfriend went to visit him today, was only meant to be a day visit but now he is talking about staying the night, he has joked to me about using ketamine with his friend just for fun. When I told him I won’t stay with him if he does any drugs especially if he tries to do it behind my back. He told me if he isn’t doing it at home and only has a little bit for fun that it shouldn’t worry me so much. Then he says “only joking”. I made him promise me he won’t touch anything but I’m having a hard time believing him. My partner is an ex drug addict, he’s grown up around people that normalise drugs and use them for “a good time” he hasn’t touched drugs in a about a year or so when we got together and now expecting a baby.

My question is: am I in the wrong for threatening to leave him over this? Is it harmless him having abit of fun with friends if he isn’t home? I’m all for having a few drinks, but taking drugs like ketamine is crossing a line imo.

I wanted to hear what others think.