Is this more likely to be a lactose or protein intolerance?

I've always been a huge fan of dairy, but for the last 10 or so years (started mid 20's) I've had to massively limit how much I have due to the issues it causes with my health. I've noticed that having too much dairy will cause a huge amount of brain fog and tiredness which lasts days, with symptoms usually hitting about 24 hours after I last consumed dairy.

Interestingly I don't get any digestion issues at all. Small amounts don't trigger it (such as a piece of chocolate), but a glass or milk or dairy filled meal will.

Lactase tablets don't help at all, no matter how many I take before eating dairy.

This is also something my mum deals with, starting at the exact same age with the exact same symptoms.