[Landlord, US, CA] need to return pet fee?

We had charged the tenants a pet fee (it is labeled as pet fee on the lease agreement) at the beginning of the lease with the understanding that it is not refundable. At the time, we were told a non-refundable pet fee can be charged in California. The tenants have left and are demanding the pet fee be returned. After some googling, I read that pet fee can be treated as upfront pet rent but also read that pet fee is treated as part of the security deposit and cannot be non-refundable.

Do I need to return the pet fee? It is past the 21 days, so if I do, I would like to return it ASAP.

And thanks for all the advice on my previous post, I really appreciate it!