I think my baby hates me

I'm a new dad, 44 years old, first kid. Little guy is 16 weeks old now. He loves his mum, as to be expected, but any time I'm left alone with him for any length of time, he immediately snaps into a screaming flailing furious rage cry that I just cannot seem to do anything to stop.

He's absolutely fine with strangers (including male ones), he's been quite socialised around various places with no problems (in fact he seems to thrive in such situations) and generally he's a happy baby.

All except for me. And it's so instant and full-pelt too. I'll be talking to him, or singing to him, or making faces or whatever and he'll be laughing away and then about a minute later suddenly just CLICK and out comes the rage monster, and from there it's just inconsoleable screeching angry crying until either his mum returns or he eventually tires himself out and falls asleep.

I cannot fathom anything I'm doing to cause it. I've tried not wearing cologne, using his mum's soaps, even a dab of her perfume. Hell, I even tried not showering one morning out of desperation just in case something about the products were causing it. I've tried being my normal beard, I've tried being clean shaven, so it's not that either.

I'm not nervous or anxious around babies generally (my mum was a foster parent for years when I was growing up so I got very used to them), I do my share of nappy changes, I bottle feed him at least once a day (unless I get stuck at work).

But this has been going on for weeks now, and my nerves are starting to get a bit fried. I'm desperate to be pulling my weight and giving his mother some breathing space when I'm not at work so she can do some things for her own sanity, and I'm really worried this is impacting on her as well because she can't settle into doing something elsewhere in the house when he's screaming like this.

I could understand it more if he was nervy/wary of everyone who wasn't his mum, or even just men, but friends and family members left alone with him for a bit don't get this same reaction - with them he's all engaged smiles, so that's the part that's beginning to make me feel really hurt and confused.

I'm open to ideas! Or even just reassurance that this WILL pass at some point, as I feel like I'm really beginning to struggle with it now.