Has anyone else found that building an Ai assistant is much easier than making it genuinely useful?

I've been experimenting with different approaches to building ai assistants recently, and one thing keeps surprising me.

Getting an llm to answer questions isn't the difficult part anymore. The real challenge seems to start once you connect it to real company knowledge.

Things like:

• Keeping answers accurate as documents change.
• Handling thousands of files instead of a handful.
• Deciding whether long-context models are enough or if rag is still the better approach.
• Reducing hallucinations without making responses overly restrictive.
• Balancing response quality with latency and cost.

There are so many frameworks and architectures available now that it's becoming difficult to know what actually works well outside of demos.

For those who have deployed A i assistants in production (or are actively building one), what has been the biggest challenge you ran into that you didn't expect?

I'm especially interested in real-world lessons rather than ideal tutorials.