TL;DR: Lowered the nut a lot, works great, considering lowering the saddle some more.
I decided to stop torturing myself and do something about the high action of my classic, since the stiff strings were getting ridiculous.
Before:
12th fret string height: both E's just under 5mm 1st fret string height when 3rd fret is pressed: just under 0.7mm
Went for the saddle first, sanded it down, got the action at 12th fret to about 4.5mm low E and 4mm high E. Night and day difference already, strings super smooth.
But it also highlighted the real problem, that it was the nut that was way too high. Where I was still measuring 0.7mm at the first fret, the normal height is apparently about 0.2