Built a text-to-SQL product, benchmark improved, but should I launch or keep improving?
I’m building a read only AI tool that lets people ask questions about their database in plain English and generates SQL. working more for enterprise grade.
I tested the latest version on two BEAVER enterprises benchmark domains:
- dw_real: 89/121 correct 73.6%
- Neutron: 504/1,017 correct 49.6%
- Across both: 52.1%
The Neutron result is the part I’m unsure about. It is improving, and the system now produces fewer wrong executed queries, but it still returns “no SQL” too often when it is uncertain.
I know these scores are not good enough for “fully autonomous analytics.” My product shows the generated SQL and is readonly, so users can review it. But I don’t want to launch something that creates false confidence or wastes people’s time.
Posted Aug 29, 2026, 1:31 PM UTC
55