Swap out drill press bearings to do light milling.

Has anyone replaced their drill press ball bearings with tapered roller bearings to do light milling on the drill press or is that just an all around dumb idea? I have a really small shop so trying to be as efficient as possible with tools. I'm just doing one off guard slots here and there and using a file is really time consuming.