Managing Big Projects with Hermes

I have been working on quite a large scale project with my agent the last week and am having a lot of trouble just getting it to stick to the spec.

We spent a long time planning out a detailed spec, did a long Q&A and tried to leave no stone unturned so the implementation was smooth.

But its been a fiasco. It keeps on going off course into random bespoke tasks and just wont stick to the project spec/implementation plan in detail despite my best efforts to keep it on track, I am having to hold its hand through every step where I want it to be a project manager.

I am sure it is something I am doing wrong so am open to advice.

Is it a model issue? On gpt-5.6-sol. Is it a memory issue? Use obsidian.

Perhaps there are in built commands I am underutilizing (i know about goal prompts but not as useful in this case).

Or are there any tools/skills out there people use to assist? I have installed superpowers and spent some time making sure those didnt clash with any existing skills I had but have found its impact to be minimal.