HR Trying to Rescind Reasonable Accommodation - Advice Needed

Long story short, HR granted Reasonable Accommodation through an informal process which said that I could work from home and not come into the office to avoid UV exposure from a long drive to and from work last year.

I continued to travel to conferences assuming that this could not be reasonably accommodated by the organization because my manager continued to schedule me for travel and because no one explicitly said that I no longer needed to travel.

(Does that somehow invalidate my reasonable accommodation?)

Now the organization is saying that because I was travelling, it is difficult for them to allow for me to continue to work from home. (which makes no sense)

My original doctor's note is very clear on the matter and says that I should work from home.

I'm fine with travelling for conferences only but not driving to and from work often because it is too much exposure. I fear though, that if I say I can travel to conferences, then they'll turn around and say that I need to come in to the office. I'm only fine with traveling because that is part of my job and I'm sure that I'd be fired if I refuse to travel or acquire a note from my doctor saying travel should be avoided (because it should).

Any advice on how to deal with HR while looking for legal support is helpful. At this point, they've moved from only having conversations to sending communications in writing.