My partner often indicates they don't want me to touch them (in any way) which I respect, then gets sad that I don't try to touch them (non sexually)

My partner has been LL for a long time and I've accepted it, even if I don't like it. I don't like our frequency but I never try and put pressure on them but I truly hate when casual physical contact gets refused by them, then when I'm careful about it that my partner gets upset because they think I'm upset with them because they feel like I'm withholding.