Can I drive car for a while with real brake shoe rubbing a bit?

The force needed to push the car in neutral is about double and i can feel one wheel dragging a bit if i let it roll. But i need to see if car drives ok till heated up fully before commiting to repairing the brakes. Am I ok or it might be worse at speed? Car is not dragging to a side or anything.