How often should I clean my grinder?

I usually clean it every time after I brew, but I’ve found this to be quite annoying and left me wondering if it’s even necessary to clean it as often as I do.