Here's how I got Spectral Fist specialist with 10:23 remaining.
In the screenshot I have Arcane Splinters. That was a complete misclick. You should only take Spectral Fist, and upgrade it with Ground Pound as soon as possible. I don't know if Spectral Clutch is good or if it sabotages getting the achievement; I was so well on my way to get the achievement that I didn't want to risk taking it.
Map: Haunted Caverns (the first map) in agony mode
Class: Sage, multiclassed into archer
Items: mark of the arrow; mask of madness; holy relic; iron ring; collar of confidence; firewalker boots; blood-soaked shirt; bloodstained wrapping (attack speed and multi-attack