Why didn't Harry stop Voldy from getting the Elder Wand?

I get that he thinks Dumby didn't want him to use the Elder Wand, but why didn't he stop Voldy from taking it?