UofM Celebrity offers LOR: Should I have accepted?

So for context, I’m an out of state student applying to the University of Michigan ED. My grandparents are amazing friends with a beloved, former head coach of Michigan Football/Basketball (trying not to dox myself too much) and his wife.

I have become somewhat acquainted with him over the years, giving sailing lessons to his grandkids and doing some minor repairs around his house as-well as taking care of his dog when he was sick and injured.

Anyway, multiple times he has offered to write me either a LOR or a referral to the admissions office. I’ve thanked him and told him that I would consider it, but ultimately decided I wouldn’t. I think it was a combination of ethical reasons, the fact that I don’t see him as an amazing recommender due to my lack of time with him, and drawing unnecessary attention to my application that led me to declining.

I feel like I have a good shot on my own, I have some solid stats, ECs, essays, recs, etc… but I still don’t know.

Either way, the offer still stands if I get deferred. Should I have and should I take it?