I've pissed people off at the hotel I'm staying at by moving their towels. oops

I've seen theses same people multiple times, what they do is they come down to the pool at like 7am to reserve loungers by leaving towels on them then they go out and enjoy their day and come back at like dinner time to lay on the loungers and go in the pool so there's people who actually need loungers who aren't getting them because they reserve them while they are at the beach all day. so when I went down I moved a towel of theirs so I could lay on the lounger because I was actually using it and I was planning on just putting it back before dinner. but I'd lost track of time and seen them coming so I got up and out it back and they went mental because I was on it all day when they weren't there but I don't see the issue. and I asked ahat the issue is since they aren't there anyway and I put it back but they were just mad for no reason. like I didn't get it dirty or wet or anything. so I asked what they were upset about and they just kept repeating "cause u used our loungers we were keeping" AND WHATT?? it would be different if I wouldn't give it back but they shouldn't be keeping them anyway when they aren't there. but now everytime they see me they glare at me and call me names just loud enough for me to hear. it's honestly childish and these are grown adults, they're being racist against me now for no reason like they're being sooo dramatic