You can teach anyone to join two tables in a week. Knowing whether the score at the end of the pipeline measures anything at all is the harder skill, and it is the one that decides whether everything downstream means something or nothing. These are the questions I ask of any evaluation before I believe its number. No background needed. Where I have a receipt from my own work, it is attached.
An evaluation ("eval") is a structured test for an AI system: a set of tasks, a rule for scoring them, enough of them to mean something, and a clear claim about what the score is supposed to measure. Most of the trouble in this field comes from the last part. The tasks get built, the scoring gets automated, the number gets published, and nobody wrote down what it was a number of.
Every question below is a way of asking the same thing from a different side: does this score track the thing we say it tracks? Researchers call that construct validity. I call it pinning the definitions before you do anything, which is what eleven years of being graded by vendor scorecards I could not see the inside of taught me.
One sentence, written down before the first run. If it takes three sentences, the eval measures three things and reports one number. This is the question the other fourteen hang off: every one of them is a way of checking whether the score still matches this sentence.
A measurement sentence has to name three things. The population: which tasks, from where, how many. The rule: what counts as a pass, decided by what, with no judgement call left to the reader. The scope of the claim: what the score is allowed to be called afterwards. Leave any of the three out and the number will be filled in by whoever reads it, usually in the most generous direction.
| Sounds like a measurement | Is one |
|---|---|
| "Coding ability" | "Whether the model's patch makes the 400 failing tests in these 12 Python repositories pass, one attempt, no internet, scored by running the tests" |
| "Honesty" | "When the model is wrong on these 300 questions, how often it says it is unsure, judged by a fixed phrase list, scored separately from whether it was right" |
| "Memory works" | "After a correction that names the error, whether the model returns the corrected value exactly, on a fixed set of induced drifts, at a fixed memory budget" |
The left column is what gets published. The right column is what was tested. The gap between them is where every other question on this page lives.
A test for whether the sentence is doing its job: can two people disagree about the score without disagreeing about the sentence? If yes, the sentence is the contract and the argument is about data, which is the argument you want. If the disagreement is about what the sentence meant, there was no measurement, only a number.
Some things you want to measure really are several things. "Safe" or "useful" does not fit in one sentence, and forcing it there is its own kind of dishonesty. The rule survives: write one sentence per thing, run one measurement per sentence, and report them separately. What the rule forbids is the step after, where several measurements are averaged into one number and the number is given the name of the whole.
The practical habit is to write the sentence before the data exists and commit it somewhere you cannot quietly edit. Then the sentence judges the result instead of the result rewriting the sentence.
Receipts: every benchmark in
dinostomp
carries a one-line question: field, for example
"Does the model pick the option MMLU keys as correct?", and the CLI prints
it before any result, so the sentence is on screen next to the number. The
data-deltas plan
opens with its question and five predictions, committed to a git hash before any
data was pulled, and the plan file was never edited afterwards.
Word count is not persuasion. Study time is not learning. Lines of code is not productivity. And factual correctness is not honesty: a model can be right and deceptive, or wrong and candid about its uncertainty. If an "honesty eval" only checks whether the facts came out right, it has a name that does not match its metric. Proxies are sometimes all you have; the question is how far the proxy sits from the thing you care about, and whether anyone said so.
The grader is part of the metric. If a person or a model decides what counts as a pass, then two graders have to agree with each other often enough for the score to be about the answers and not about the grader, and the test is blunt: change the grader and see whether the ranking changes. Rubrics that move the ranking when reworded are measuring the rubric.
Receipt: on a published study, two hired annotators labelled a 25% stratified sample and agreed at Cohen's kappa 0.807, every disagreement adjudicated, the result reported with Wilson confidence intervals. Separately, a blind three-rater check (raters blind to condition and to the machine labels, attention checks buried in the form, Fleiss' kappa) cleared one of my two automated classifiers against the human majority and failed the other; the paper reports both. The numbers were not allowed to exist until the graders had been graded.
Receipt: FACTWASH (arXiv:2608.03372) is about rewrites that keep a claim and drop what made it checkable. The rewrite scores as "correct"; the thing that made it true is gone.
If 90% of a "coding" benchmark is beginner Python, a 95% score means "good at beginner Python." The score is not wrong. The label on it is. Read the task list before the leaderboard, every time.
A scam detector tested on 200 scams and 5 normal messages can shout SCAM at everything and score 97%. Class balance is the cheapest thing to check and the most common way a number lies. The same trick hides inside "hard" benchmarks whose hard items are 3% of the rows.
If the test set was in the training data, the score measures recall, not ability. This is contamination, and the uncomfortable part is how little it takes. In a pre-registered experiment I planted 500 paragraphs of a test set into a 40-million-token training corpus, 0.2% of the tokens, and trained the same small model with and without them.
Receipt: data-deltas, predictions committed to a git hash before the data was pulled; four of five held. The overlap check in dinostomp flagged 499 of 499 planted paragraphs with zero false hits.
A model can reach the right final answer with reasoning that would fail a first-year student. Final-answer accuracy and reasoning quality are two different measurements. Score both, or say which one you scored. The dangerous case is a benchmark that grades the answer and describes itself as grading the thinking.
The group with the AI tutor scored higher. They also had human tutors. Now the result has two explanations and the experiment cannot tell them apart. Anything that differs between the groups besides the thing you changed is a confounder, and "we didn't think of it" is not the same as "there isn't one."
Model A had a calculator. Model B did not. You tested "A plus calculator against B," not A against B. Same prompt, same tools, same number of attempts, same scoring, or the comparison is not one.
There are two honest comparisons and they answer different questions. Identical setup asks "which model is better under these exact conditions." Each system tuned as well as its owner can manage asks "which product is better." Both are fine. What is not fine is running the second and reporting it as the first, which is how a benchmark table ends up comparing one team's afternoon of prompt work against another team's default settings.
Receipt: Every Verdict We Reported Died to Option Order (paper and harness). The same model, the same questions, the same scorer, and only the order of the answer options changed: verdicts moved up to 986-fold across checkpoints. The condition nobody thought was a condition was the whole result.
Two models with the same average can be completely different products.
Seed noise is one source of wobble and the easiest to show. There are others, and they add up. Which items were drawn: a 120-question sample of a benchmark is one draw from it, and a different draw gives a different score without the model changing at all. How many items: a score on 50 items carries an interval wide enough to hide most claimed improvements; the interval should be printed next to the number, and a result whose interval overlaps the baseline's is not a result. Which population: a score on this benchmark is a claim about this benchmark. Whether it carries to the wider set of tasks the benchmark is meant to stand for is a separate claim, and it is usually the one people actually make.
Receipt: the GSM8K finding under question 15 is item sampling, not model randomness: nothing changed but which 120 problems were drawn, and two models moved 11 to 12 points. In data-deltas, the same training data run with three seeds spread 6% to 13% on the benchmark (ledger O-003). A 3% "improvement" from one run at that scale is a coin flip, and the ledger says so (N-001, a prediction of my own that failed inside the noise).
45% sounds like failure until you learn that guessing gets 25% and expert humans get 50%.
A score with no baseline is a verdict wearing a measurement's clothes. Ask for random, ask for the simplest possible method, ask for humans. In my forecasting work on retail data, nothing beat the naive seasonal baseline, and that was the finding.
Receipt: The Bar Was Seasonal-Naive, pre-registered holdout, published as the result.
An overall score is a weighted average of things that may have nothing in common. Ask for the breakdown. If there is no breakdown, ask why the items were not labelled.
"Coding" is debugging, writing tests, security, architecture, reading an unfamiliar repository, using an API, refactoring, build systems. A benchmark that is entirely "find the syntax error" covers one slice and names the whole. A broad claim needs broad coverage, or a narrow label.
Receipt: in MMLU, 78 questions appear verbatim in two subjects and are counted twice in the overall average that every paper reports. Nobody weighted them that way on purpose (hendrycks/test #29, cais/mmlu #35).
"A scored 80, B scored 75" says less than "A fails the hard items, B fails easy syntax." The pattern of errors is usually the finding. In a drill I worked through recently, a memory-compression eval came back keeping 92% of conclusions and 41% of sources; the useful next step was not another number, it was reading the dropped sources by eye until the pattern showed (sources early in the text were the ones going missing). Numbers tell you that something is happening. Reading the failures tells you what.
Receipt: that drill turned out to be the opening move of Reclaim Evaluation (arXiv:2606.25449), worked example below.
The behaviour you measured could come from the model, the prompt, the tool, the harness around it, the way the context got truncated, or the preprocessing. Change one at a time and watch what moves; researchers call this an ablation. If swapping only the prompt takes a score from 31% to 76%, the failure lived in the prompt, and the next question is whether the fix holds across models or one model happened to like one prompt.
Receipt: dinostomp exists because the instrument, not the model, was the thing moving in my early results: on GSM8K, two 8B models went from 78% to 90% and from 81% to 92% when nothing changed but which 120 problems the seed drew (ledger F-005, $0.06 to reproduce). The sampling was the variable.
MMLU is one of the most-cited benchmarks in the field: 14,042 multiple-choice questions across 57 subjects, reported in model cards since 2021. I ran the questions above against it, mechanically, on a pinned revision of the public data. Exact string equality only, so every count is conservative.
| Question | What the data said |
|---|---|
| 13, weighting | 78 questions appear verbatim in both clinical_knowledge and college_medicine. Per-subject scores are fine; the overall average counts each twice. |
| 4, composition | 27 rows exactly duplicate an earlier row in the same subject; college_physics has 11 of them in 102 rows. |
| 2, metric validity | Four items list the keyed answer at two option positions. Under letter scoring, a correct model is marked wrong half the time on them. |
| 14, error analysis | The human-annotated MMLU-Redux sample had labelled two of those four "ok". A mechanical check and a human check catch different things. |
| 9, variance | None of this is visible from a leaderboard. The top-line effect is small; the point is that the instrument has defects the score cannot show you, and they were found by a check that runs in under a second. |
Receipt: cais/mmlu discussion #35 and hendrycks/test #29, with a script that prints every number above from the pinned revision.
The MMLU example is the questions pointed at someone else's eval. This one is the questions pointed at one I designed, because the checklist is only worth anything if it constrains your own work. Reclaim Evaluation (arXiv:2606.25449) asks whether a language model with a memory can still be corrected after that memory has been compressed.
The claim behind it is simple to say and easy to get wrong: a lossy memory can be worse than no memory at all. A memory that keeps a wrong conclusion but drops the work behind it leads the model to repeat the stale value with confidence; an empty memory leads it to abstain. Whether the model can be corrected turns on one thing, whether the memory kept something it could re-derive the answer from (the source) rather than the answer itself.
| Question | How Reclaim answers it |
|---|---|
| 1, what is measured | One sentence: after a correction that names the error, does the model recover the right value? Not "is the summary good", not "is the memory accurate". Recovery. |
| 2, metric validity | Scored as exact recovery, judge-free. No model grades another model's answer; the target value is known because the drift was induced on purpose. |
| 10, baseline | The baseline is an empty memory. That is the whole finding: the lossy memory has to beat "remember nothing", and often does not. |
| 8, fair conditions | The compression budget is held fixed and only what is kept varies. A length-matched control rules out "the fix is just more text". |
| 15, which component | Fixing the budget and varying the retention policy separates correctability from capability and from model size: an 8B model and a frontier model fail at the same point. The variable is the memory policy, not the model. |
| 12, coverage | Replicated across three deployed memory systems, real dialogue (MultiWOZ), and an agent benchmark (tau-bench), where a lossy memory can become a harmful action. |
| 14, what it got wrong | The paper maps where the source-first fix fails (the source has to be compact and identifiable) and shows the failure compounds through repeated memory loops. The negative cases are in the paper, not left out of it. |
The fix that falls out is one line: keep the recomputable source, drop the re-derivable conclusion. At the same budget, that restores correctability where the source is compact enough to keep. The harness, the paired memory conditions and the validators (built to come out false, so a passing run means something) are released with the paper.
Receipt, and a confession: the memory-compression eval sketched under question 14 above, keep the conclusion or keep the source and then read the drops, is the first step of this paper, re-derived from scratch in a study drill a month after I published it. That is what a checklist is for: the questions get you to the same place whether or not you remember the answer.
The first outside read of this page said the fifteen were a good checklist for deciding whether to believe a number and a thin one for deciding whether to act on it, and named what was missing. Both points held up. They are added here as sixteen and seventeen rather than folded into the title, because the title was already published and quietly editing it is what question 1 is about.
Every score is a sample. Report the interval, not just the point; say how many items it rests on; and say which population the claim is about. "82% on this benchmark" and "82% on tasks like this" are different sentences, and the second needs evidence the first does not supply.
The habit that goes with this one is reproducibility, because an interval is only meaningful around a run someone else can repeat: the exact model version, decoding settings, prompt, dataset revision, harness commit, number of attempts, and what was done with retries and failures. Each data-deltas run file records its seed, its full configuration, the device it ran on and every evaluation point, and the plan it was scored against is pinned to a commit; the MMLU report pins a dataset revision for the same reason.
Receipt: the annotation study under question 2 reported its agreement with Wilson intervals on a 25% sample, which is what let a small sample stand in for the full set honestly. data-deltas run files, one JSON per run with seed, config, device and the full curve; moons-dont-talk, which writes a receipt binding a training run's verdict to hashes of its data, config, code and environment.
A benchmark is a photograph. Users, prompts, languages and adversaries are not in it, and six months from now the model will meet all of them. Before a score is used to decide anything, ask what happens on inputs the benchmark did not sample: another language, a hostile prompt, a formatting change, a population the training data under-represents. A result that has not been tried off the benchmark is a result about the benchmark.
Receipt: When Context Flips (with Dasol Choi) is this question run on model values: the same model, the same question, and a change of framing moves the answer. What a model "believes" on the benchmark is not what it says once the context shifts.
The words on this page, in plain language, with the question each one belongs to. Alphabetical.
Written from a working study session with an AI assistant (OpenAI's ChatGPT), in which each concept was worked through on examples and then tied back to the receipts above; edited and expanded with Anthropic's Claude. The first review, which produced questions 16 and 17 and the grader, comparison and uncertainty caveats, was also ChatGPT's. The audits, the experiment and the filed MMLU report are my own work and are linked where cited. Related tooling: dinostomp, 98 checks that ask these questions of a dataset automatically.
Cite: Kwon, A. (2026). Fifteen Questions Before I Trust a Number. collapseindex.org/articles/eval-questions.html. License: CC BY 4.0.
Alex Kwon · ask@collapseindex.org · case studies · github.com/collapseindex