HarnessSQL trains small Qwen3 models inside their SQL agent harness

An arXiv paper reports Qwen3-14B rising from 22.2% to 54.8% on Spider 2.0-SQLite when trained in the same harness it runs in at deployment.

A paper posted to arXiv on October 8, HarnessSQL, argues that text-to-SQL models fail as database agents because of a train-deploy mismatch. They are trained to write one static query, but deployed agents inspect schemas, run probe queries, diagnose errors and revise, inside a harness that only shows up at inference time. The authors train inside that harness instead. They report that on Spider 2.0-SQLite, execution accuracy rose from 15.5% to 45.2% for Qwen3-8B and from 22.2% to 54.8% for Qwen3-14B. The practical point for anyone building a database agent is to evaluate it in its real loop, not on static queries.

According to the abstract, the method builds isolated, executable databases with hidden answer checks. Teacher models run inside the target harness, and only verified trajectories are kept for supervised fine-tuning. Execution-reward RL follows. The authors say the gains transfer to BIRD-Interact and LiveSQLBench, but the abstract gives no numbers for those. On X, elvis (@omarsar0) highlighted the result on October 10.

What's still unclear: we found no released code, environments, checkpoints or license. We also found no comparison with frontier models run in the same harness, no detail on the harness design, and no outside reproduction.

The score sits on the SQLite subset. Top Spider 2.0-Snow entries reach 96.7%, per a separate paper citing the leaderboards as of September 24, but one research write-up notes that Spider 2.0's gold answers are public for self-evaluation. That makes cross-system comparisons shaky.

Most readers won't run the RL step. The cheaper takeaway is to test your agent with real tool access. One example is a proposed open-source design that gives a prompted model read-only probe queries.

Sources