Neighbor has been harassing me about smoking tree on my property

I’m not sure if this is the right thread to be posting this in but I have a neighbor that moved in recently (I have lived here for 5+ years and been smoking on my property the whole time without a problem) They live right next door and while the houses are “close” together, we live in the city so that’s expected. it’s not the nicest area either as there is a lot of drugs and homeless in the area as well. But recently my neighbors have been watching me and or coming up to me and complaining about me smoking in my yard and have asked me to do it somewhere else. They have said they don’t want to call the cops but will if they have to. My question is what do I do in this situation? Everytime I look into it it says you can legally smoke weed on your own property but do not want a pointless issue with people i don’t even know. If anyone has ideas or suggestions I’d love to hear them, Thank you