ChatGPT keeps breaking our agreed rules & workflow

I own some faceless YouTube channels and I create educational/documentary style videos. ChatGPT handles everything prior to video generation. Script, images, workflow etc.

Over the course of making a few videos, ChatGPT made a lot of mistakes so it suggested we build a strict workflow almost like a checklist so it always knows what to do and what not to do. The checklist will contain a Phase plan (Phase 1, Phase 2 etc). Image specs. Branding info about the channel etc etc.

Now here is the problem. It usually follows the workflow perfectly for the next few scenes. Then randomly, it’ll completely break our rules and generate an incorrect image. It’s either against our agreed image specs, or it goes against the branding of the channel or it’s just outright completely wrong. It then explains that it broke our rules and will try again. It ends up taking a long time with a lot of back & forth until it finally understands how to get back on track with our rules. The cycle then continues. This doesn’t only happen with images. For example, Phase 4 is image generation. All Phases before that are planning in text mode. Sometimes at the end of Phase 2, I’ll say “proceed to phase 3” and it’ll start generating the image (which is Phase 4). I will stop it and it will say something like “😂 I broke our rule again. I skipped Phase 3”.

This is causing me a lot of delays in production and is driving me insane. Does anyone have any tips? Is this simply just unavoidable? Is there a better system to ensure GPT always stays on track with our agreed rules/workflow? Any help would be greatly appreciated! Thank you