Should I cancel my credit card ?

I got this credit one card with Amex( I know) during a time I needed to rebuild my credit. Only had it for about two years and the limit on it is only 500. I got the annual fee charged and they said they can’t do anything to waive it. I’m just over this card, I don’t ever use it and when I do I pay it off. They offered me a Credit increase but have to pay 14.95 to accept it. I’m wondering if I cancel it how much will that affect my credit score?