Question about canceling a secured credit card

Years ago my credit situation was pretty horrible so I obtained a secured credit card for $200 to build my credit. Since then my credit rating has increased substantially and now I have another credit card with a much higher limit that isn't secured through the same Capital One.

I don't really use my secured credit card anymore but because it has been open the longest out of all my credit cards would it make sense to keep it open despite paying the annual fee every year?

I hate paying the $80 a year for the maintenance fee but I don't want to cancel it and screw up my credit too bad.