Everyone shares benchmarks.

Fewer people talk about what happens after launch.

After spending the last year deploying and maintaining LLM-powered applications, these are the lessons that stuck with us.

1. The model is rarely the biggest problem

Most production issues came from everything around the model:

• retrieval
• data quality
• latency
• monitoring
• user experience

Changing models usually wasn't the biggest win.

2. Users don't care which model you're using

Nobody has ever asked us if a response came from GPT-4, Claude, or Llama.

They care about three things:

• Is it fast?
• Is it accurate?
• Can I trust it?

3. Better prompts don't fix bad systems

Prompt engineering helps.

But it won't solve:

• incomplete context
• stale documents
• poor retrieval
• missing business logic

A good system beats a clever prompt almost every time.

4. Observability is harder than expected

Debugging traditional software is already difficult.

Debugging why an AI answered the way it did is another level entirely.

Logging prompts, retrieved documents, latency, and outputs became essential.

5. Edge cases never stop

Every production deployment introduces questions no one tested for.

People ask things differently.

Data changes.

Requirements evolve.

LLM applications need continuous iteration, not a one-time launch.

6. Simplicity scales better

Some of our best-performing systems weren't the most sophisticated.

They had:

• fewer moving parts
• clear retrieval logic
• reliable guardrails
• predictable behavior

Simple systems were easier to improve over time.

Our biggest takeaway

Production AI isn't really about the model.

It's about building a reliable system around it.

That's where most of the engineering effort ends up.

For those running LLMs in production, what lesson did you only learn after shipping?