What's something about a person that can make them hard to love?

Love in casual or romantic terms just meaning hard to wanna be around them, spend quality time around them, just prioritze them in general.