If removed as an authorized user does it affect credit score?

Wife and I have a credit card with some debt on it, it’s hers but I’m on there as an authorized user. We did a balance transfer to it and it knocked my score down about 15 points. If I’m removed will that matter? Or is the damage already done?