Data agents still miss real analyst work, and a semantic layer helps

Shreya Shankar's Data Agent Bench tests agents as junior analysts over messy multi-database data; a semantic layer significantly raised accuracy.

If you're pointing an agent at your company's databases, expect to review its answers. In a dbt Analytics Engineering Roundup episode (October 23, 2026), Shreya Shankar tells Tristan Handy that the best agent in her Data Agent Bench (DAB) passed 61% of queries on the first try. Agents given a semantic layer did significantly better.

DAB tests whether an agent can do a junior analyst's job across multiple databases, messy data, tribal knowledge and free-text columns. The arXiv paper describes 104 queries across 17 datasets and 4 database systems, drawn from a study of enterprises in six industries. Its abstract gives a different best figure, 57% pass@1. The episode doesn't explain the gap.

The public DAB leaderboard now lists much higher scores, up to 0.9628 pass@1. Most top entries are flagged as using a "tuned prompt" built from close study of DAB's task conventions, and they also used hints. So treat those numbers as what is possible with heavy, benchmark-specific setup. They are not what an agent will do out of the box.

Shankar also names two failure modes worth watching. Agents tend to select * from two databases and join them in pandas instead of pushing computation down to each database. And you can't write a good LLM-judge rubric until you've read many outputs yourself, which she calls criteria drift.

One alternative benchmark, Argo-Bench, grades agents on the simulated consequences of their actions in a 235-table ERP warehouse. There, the strongest of 14 models scored 95 or higher on only 34.8% of tasks. The episode doesn't say how large the semantic-layer gain was or which semantic layer was used.

Sources