I refused to sponsor a colleague doing a charity event, now other colleagues are going the same. AITAH?

I work in a medium sized office, there's 20ish people and we would be pretty well paid for what we do. That is just to say, it isn't a financial concern why I or my colleagues refused to contribute.

I work with a guy in his mid 20s who we will call James. James is doing a sponsored skydive for a charity next month. James has sent an email around the office about it, and yesterday put up leaflets in the canteen and bathrooms with his revolut details. A few of us (James not included) were talking over lunch and the matter of whether people had sponsored James came up. Someone brought up that James had been away in May, and had done skydiving then, so he was really excited to do another one.

I wasn't happy about this, because while I like it when people are raising money for charity, the first €300 of the donation goes toward funding something he really wants to do, and I would much rather donate directly to the charity so they see all of the money, or at least sponsor someone for doing something that pushes them out of their comfort zone or is a challenge to complete like a run.

I said as much to my colleagues at the table, and they agreed with me that it's pretty poor that the people who are benefiting the most from this are James and the skydiving company. Some said they weren't going to donate to James and they will donate directly to the charity instead, and if he gets pissed off then it'll show he's really only doing it for the experience and kudos.

Now I am concerned that I will come across as the asshole if James finds out I have started this as the reason he doesn't get sponsored at work.

TLDR: Work colleague wants to do a skydive, so is asking colleagues to subsidise his fun experience under the guise of raising money for charity, with the donations covering the cost. AITAH?