Is it rude to trail someone on an mx track?

Just a rant of an event today that’s living rent free in my head. I try not to ask dumb things on reddit or try to have my feelings justified but today I had someone get mad at my for trailing him and just don’t know if it was justified.

Context- fun day out at the track with my wife and son. I’m older (35) and want to try to get myself into a racing next year so I’m trying to get better (used to ride as a kid/teenager but stopped until this year). Was the end of the day and most people had left. Went out for one last moto and about two laps in my arm was cramping so I pulled off to get water. The only other guy on a full sized mx bike gets on the track as I’m getting off. I thought “oh shit maybe he was looking to ride together”. We rode a few laps earlier in the day where I would try to keep up. So I drank some water and hopped back on the bike. This guy was pretty fast. Had great technique and numbers on his plate so I assumed he was a racer. Also had a go pro on earlier in the day. I thought “oh cool I’ll wait for him to pass me up and I’ll see if I can keep up” so he passes and I gave him a nod and I got on it. Now the whole time I was not aggressive AT ALL. Never once did I try to pass him ,cut him off or anything that would signal aggression. The only time I was even close behind him was when I started and this one section when I would catch up a bit but he would gain traction right after that turn. Went about 3 or 4 laps and I fell over on a berm. I putted around for a lap till he passed me and got on it again. I was thinking the whole time how I was going to go up to him after and tell him “man I was trying to keep up but couldn’t do it and just kept making mistake after mistake”. Went about two laps and he slows down at the part where you get off and SHOOTS ME THE FINGER. (I matched his speed to to let him get off safely). I was like “what is that about?” So I get off and ride up to him and was like “hey man what was that about”. He take his helmet off pissed off and yells “WHY WERE YOU ON MY ASS THE WHOLE TIME, I FUCKING HATE THAT SHIT” I’m like puzzled because I wasn’t really on his ass but maybe certain parts and everything I did was safe and never was putting him into harms way. He goes on about how he rode 4 laps then I waited for him (idk why he said that maybe saying i was trying to tire him out before going?) and is on his ass and then I disappeared and came back on his ass. I’m telling him “dude my bad you’re the only one out here I thought maybe you wanted to ride with someone or something , also I fell and tried to catch up to you again” I told him he was the fastest guy out here and I am trying to get better and trying to learn. He eased up a bit when I said that and he said he was sorry but he just hates when people do that (still kinda being a dickhead). I rode away back to our setup.

Is it common courtesy to not try and race people at an mx track? I was honestly eating so much roost and was still just having a fun time but I guess he wasn’t. Just feel like he could’ve done things to let me know he didn’t want to ride around each other. Like maybe let me pass him or just gone to a different part of the track. But it honestly seemed like he was into it or I would’ve just backed off, it didn’t have to end how it did. Should I ask people if they want to try to play catchup before hand? Never had someone get pissed at it before