2nd tri - feeling like people are getting to me

I’m 23 weeks this week. I’ve been lifting weights for a few years and have a very active job where I’m on my feet all day walking/pushing/pulling etc. I’ve been pretty consistent with my workouts (lifting 3x a week) with some slight modifications. I don’t lift super heavy to begin with but I have lowered my weight some, and try to breathe through the exercise rather than use valsalva during things like chest press.

I do get warm and sweaty, my heart rate does get up to the 160’s during my sets but settles down nicely during my rest times, I do try to listen to my body… but it’s so hard when there’s so much external noise.

My coworkers are always yelling at me not to lift the “heavy” 30 lb boxes we have to move telling me I’ll hurt my baby or cause a stillbirth. I feel like some of my gym regulars give me the side eye as my belly gets more noticeable. My doctor just says to do whatever I did before and modify for my comfort.

I feel like other people’s paranoia is seeping into me now and every time I go to the gym and get a good sweat on, I feel good but have this thought in the back of my mind that they could all be right and I could be harming my baby. I’m so relieved when I feel him kicking afterwards, that everything is likely okay in there.

Anyone else been through this? I was so confident and comfortable before but have been more nervous as of late.