26:51
New session w/@sh_reya where she goes over a useful new eval called...
Hamel Husain@HamelHusain
31 views
Aug 14, 2026
~1 min read
New session w/@sh_reya where she goes over a useful new eval called the Data Agent Benchmark (DAB).
Data agents answer business questions like “which cohort had the highest churn?” that a data analyst would normally answer.
DAB recreates the mess of a real data warehouse. Each task spreads data across at least two database systems, with inconsistent join keys, key values buried in free text, and ambiguous or ill-defined schemas.
Timestamps from the talk:
00:00:52 What a data agent actually is
00:01:15 Why big companies build their own
00:04:04 The 4 challenges that break enterprise data 00:07:19 Inside DAB: the benchmark
00:11:11 How frontier models actually score
00:13:36 The five ways agents fail
00:18:05 Why planning is the biggest failure
00:21:53 Can you cheat the benchmark?
00:23:03 The semantic layer strategy
00:24:35 Getting agents to clean messy data
00:26:39 The final takeaway
Data agents answer business questions like “which cohort had the highest churn?” that a data analyst would normally answer.
DAB recreates the mess of a real data warehouse. Each task spreads data across at least two database systems, with inconsistent join keys, key values buried in free text, and ambiguous or ill-defined schemas.
Timestamps from the talk:
00:00:52 What a data agent actually is
00:01:15 Why big companies build their own
00:04:04 The 4 challenges that break enterprise data 00:07:19 Inside DAB: the benchmark
00:11:11 How frontier models actually score
00:13:36 The five ways agents fail
00:18:05 Why planning is the biggest failure
00:21:53 Can you cheat the benchmark?
00:23:03 The semantic layer strategy
00:24:35 Getting agents to clean messy data
00:26:39 The final takeaway
The table below shows how DAB compares to related benchmark categories.
Link to YT: youtu.be/ubk57rW_KUo
Link to the notes: hamel.dev/notes/llm/ai-p…
Link to YT: youtu.be/ubk57rW_KUo
Link to the notes: hamel.dev/notes/llm/ai-p…
