Is it petty to keep track of lunch money with a coworker?

My coworker and I have been splitting lunch orders for months. We usually alternate who pays and it’s always been roughly even.

Yesterday I realized I’ve probably paid a lot more than him over time, so I asked if we could maybe start tracking it or just pay separately sometimes.

He got weirdly defensive and said keeping track of small things like that is “petty” and that I’m making it awkward.

Was I wrong for bringing it up?