How much matter would we have to move to the moon for it to fall into us?

If we started colonizing the moon, we'd likely have to bring our own stuff to the moon for materials. How much could we bring to the moon before the increase of mass would cause it to fall into us? Would that even happen? At any point would the earth and moon orbit around a central point, like Pluto and Charon do?