Why isn't there a way to prevent someone from joining random duties?

So there is the mute list and block list. I wouldn't have to see someone's chat if I didn't want to, but what if someone was such a jerk, I didn't want to even be in the same group as them?

Right now my only recourse is to leave and find another group, possibly inuring a leave penalty. I do think blacklists prevent someone from joining a PF, but I believe they do not work for DF.

I would love a button that was like "I do not ever want to be in the same group as this person"