Do animals (mainly birds and deer) abandon their young sometimes after direct human interaction?

I remember hearing when I was younger (about 15 years ago) that such animals are more likely to abandon their young after petting it for too long and leaving a foreign scent.

I've heard more recently that theory had been rejected and that is not actually true?

..Pulled a baby bird out of a lake out fishing recently and was hanging out with it for a good ten minutes before finding a safe spot to let him off. My buddy was telling me that I shouldn't touch it too much so that its mother doesn't abandon it.