Possible to give someone Edit permissions for document library, but not ability to remove members from site?

I have given Contribute permissions to a test user in Sharepoint, and found they can remove other members with the same permission level. I thought Contribute prevented the ability to manage members? Or does edit permission grant this ability?