Idle balls - feed, or just upgrade?

For the balls you don't use at all, is it worth it to feed them? Or is it just better off to upgrade their weapons and their star level when you get enough frags?