How many times can you apply surface to a ball?

Like, hitting it with sandpaper removes material from the cover stick. At least theoretically, in my mind. How many times can you do that before the ball is smaller than its supposed to be? Is there a minimum diameter/ circumference requirement for a ball?