Picked up Jacob's password in covenant then went to sleep now entire town of Covenant is aggressive towards me

exactly as the title says, I got in the house stole Jacob's password and no one attacked me. I went to sleep to save my game because I'm on survival and now everyone is aggressive towards me. Is there anyway to fix this?

EDIT: I ended up just killing everyone