How safe it is to invite sex workers at home

Thinking to call her at home when ai am alone. But I always feel that once i invite her home , as she knows my address already she can misuse it or can come to blackmail. Need some clarity or did anyone do this before. Scared bcoz I live with family.