The fine-tuning advice you're getting is probably wrong...

The fine-tuning debate is exhausting.

One camp swears by prompt optimization because it's fast, cheap, deploys in minutes. The other insists you need real weight fine-tuning, train the model properly, accept that it takes time.

Everyone's yelling past each other because they're solving different problems.

Here's what I learned after debugging agents that "worked in demos" but completely fell apart in production:

Prompt fine-tuning fixes behavior. Your model returns inconsistent formats? Too verbose? Ignores instructions? That's a prompting problem. Automated prompt optimization finds what works in 10-20 minutes. I've seen teams spend weeks manually tweaking prompts when they could've just tested 200 variations automatically.

Weight fine-tuning fixes knowledge. Your model doesn't understand your industry jargon? Hallucinates product details? Fails on domain-specific edge cases? No prompt will teach it that. You need to actually train on your data. Takes 30-90 minutes, but it's the only thing that works for knowledge gaps.

Most agents need both, but not at the same time.

The workflow that actually works: Start with prompt tuning (fast, cheap, fixes 70-85% of issues). Deploy. Monitor what still breaks. If failures are systematic and knowledge-based, upgrade to weight tuning. Keep iterating because production is never static.

Real example: Customer support classifier. Problem was inconsistent output format—sometimes "billing", sometimes "BILLING_ISSUE", sometimes full sentences. Spent hours writing better prompts manually. Still failed 20% of the time.

Automated prompt optimization across 5 models? Fixed it in 8 minutes. 98% consistency. That's because it was a behavior problem, not a knowledge problem.

Different example: Legal contract analyzer kept confusing "indemnification" with "limitation of liability." Tried detailed prompt engineering with definitions. Marginal improvement.

Fine-tuned on 500 labeled contract clauses? Error rate dropped from 30% to 5%. That's because it was a knowledge problem. The base model didn't understand legal semantics.

The expensive mistake everyone makes: Jumping straight to weight fine-tuning for problems that prompt optimization would solve in 10 minutes. Or stubbornly trying manual prompts for knowledge gaps that fundamentally require training.

I wrote a blog about what each approach actually does, when it works, when it's overkill, and the workflow that successful teams actually use in production.

Work in agent reliability at UBIAI, so obviously biased, but happy to answer questions about specific failure modes if anyone's debugging production issues.