Thralls moving around my base

Hey guys, so I'm playing the Legacy (Xbox) version in a friend server, but some weird stuff is happening:

A good chunk of my thralls (specially Smiths) tend to walk around my base even if I assign them to a crafting station, they will interact with it for 1 second and start walking around my base for some reason, they tend to try get to my bar I set upstairs and even with the door locked for thralls they will managed to do it, any advice on how I can stop this?

My friend has the same issue as well