Isn't it risky that all of the Bridgerton staff know Sophie as a maid?

Now that I think about it, any other servants she has interacted with would also know about her. Wouldn't she always face the possibility of blackmail? What am I missing? Am I just thinking too much about this? Lol