Definition error
The AI uses a plausible but wrong business definition.
AI-Native Analytics · Module 10
Evaluate the AI.
The more capable analytical agents become, the more valuable it is to know how to test them systematically instead of trusting fluent output.
Core idea
A fluent answer can be wrong in three different layers: data logic, business meaning, or inference.
01
Context
The query is valid. The conclusion is still wrong.
“The numbers are technically correct for the rows it selected — but the comparison itself is invalid.”
Evaluation is broader than checking syntax. A query can execute perfectly while using the wrong denominator, an immature cohort, a duplicated join or a misleading comparison.
The analyst needs a repeatable error taxonomy and test cases that expose these failure modes.
02
Error taxonomy
Audit the answer layer by layer.
Data error
Join, grain, filter or timestamp logic corrupts the number.
Inference error
The observed pattern is described as causal or certain.
Fabrication
Fields, events, explanations or evidence are invented.
Rule
Evaluate the claim and the query separately. Correct SQL does not guarantee a correct business conclusion.
03
Challenge
Break an AI answer before leadership sees it.
Exercise · 15–25 minutes
Red-team the result.
- What denominator did the AI choose and why?
- Are compared cohorts equally mature?
- Could a join multiply the base entity?
- Which claim goes beyond the evidence?
- What edge case would make the query fail silently?
- What small “golden question” could you use to test this behavior again later?
04
AI assist
Make the model critique its own answer.
Adversarial review prompt
Audit the analysis below as if it could influence a product rollout.
Do not defend the existing answer.
Review separately:
1. Metric definition
2. Population and denominator
3. Time/cohort comparability
4. Grain and join cardinality
5. Missing or late data
6. Observational vs causal language
7. Claims unsupported by returned data
For every issue:
- state the risk,
- propose a test,
- explain what result would invalidate the conclusion.
Do not invent missing schema or data.05
Validation
The AI evaluation checklist.
01
Can I reproduce the result with an independent simpler method?
02
Have I tested known edge cases and null behavior?
03
Are cohorts and comparison windows equally observable?
04
Did I inspect the generated query rather than only the result?
05
Are unsupported explanations removed from the narrative?
06
Would this same test catch the error again in a future agent version?
PRACTICE THIS NOW ↓
Interactive case · L07
AI SQL Red Team
An AI-generated query runs and returns a polished answer. Run validation to uncover three analytical bugs.
06
Worked takeaway
Evaluation is a product, not a final glance.
Keep a small set of known-answer business questions.
Test empty groups, duplicates, late data and unusual dates.
Trace every narrative claim back to returned evidence.
Rerun evaluations when prompts, models or schemas change.
Analyst habit
When an AI answer looks unusually clean, become more curious, not less. Plausibility is not validation.