WIBTA for making my coworker with a broken foot move?

Alright, I have a coworker we’ll call them “Jim” who just broke their foot and instead of working up patients is now doing “admin duties”. Well I’m away on a trip and found out they kicked Sam (the usual who uses my desk when I’m away) out to sit at my desk because theirs makes their foot dangle and it’s bad. WIBTA for telling them they can’t have my desk when I get back? Like I feel bad but it’s my area with all my stuff and they still have less responsibilities than me even with the new status…