Error bars on an LLM eval

A benchmark score is a measurement, and like any measurement it has a distribution. Item response theory, the statistics behind standardized tests, explains why. Each question has its own difficulty, each test-taker has an underlying ability, and whether a given answer comes out right is a chance event that depends on both. A score is then an estimate of ability with two kinds of chance in it: which questions happened to be on the test, since another sample of equally good questions would give a somewhat different score, and how each answer happened to come out. An LLM benchmark has the same structure, with the model as the test-taker, and the second kind is plain to see: the same model can answer the same question differently from one attempt to the next.

So the gap between two models' scores is an estimate too, and it needs an error bar before it means anything. Evan Miller's Adding Error Bars to Evals (2024) sets out how to compute one for LLM evals. It treats an eval's questions as a sample from a larger pool and splits each question's score into an expected part and a random part, which separates the same two kinds of chance. From there it recommends comparing models question by question, adjusting for questions that come in groups, resampling answers where that helps, and sizing an eval with a power analysis. I applied those methods to one benchmark on one GPU in my homelab, and added one measurement the paper doesn't make: how much the inference server itself varies on identical requests.

The benchmark was IFEval: 541 prompts, each carrying one to three instructions that a program can check, such as a word limit, no commas, or wrapping the answer in quotes. A prompt counts as right only if every instruction in it is followed, and a model's score, IFEval's prompt-level strict score, is the share of prompts it gets right. I compared two open models that already run on my homelab's 16 GB RTX 5060 Ti, Qwen2.5-7B and Gemma-4-12B, each in the compressed (quantized) form it runs in there. Gaps between scores are in percentage points.

Each run was one job on the homelab's general GPU queue (Argo Workflows and Kueue). Before each run, a launch script claimed the homelab's GPU window, which keeps the language models it serves on demand from waking until the window is released. The job started a private vLLM server, the widely used open-source inference server, and sent the prompts through it in two or more passes, recording the server's metrics once a second. A rerun here means a later pass through the same server. Six runs counted: for each model, a main run with randomness off and a sampling run with it on, plus two Qwen runs, one varying the load, the number of requests sent at once, and one testing a determinism setting. Two more were discarded (part 6).

The short version

• On IFEval, a gap under about 6 points between models like these can't reliably be told apart from chance. Most of that error is the luck of which 541 prompts the benchmark happens to contain.
• Running the same eval twice gave different results. An identical rerun flipped 7% of Qwen's prompts between right and wrong, and batching caused it: one request at a time, Qwen's reruns matched exactly. Gemma's 2% came from different batching, so it isn't like for like.
• Rerun noise is a small part of the error bar. Averaging it out over many passes would shrink the error bar on the gap only from ±4.0 points to ±3.5. Making the server deterministic would make results repeatable and leave the error bar about as wide.
• With vLLM's batch-invariant mode requested, reruns matched exactly, though nothing confirmed the mode had turned on.
• IFEval's official grader uses unseeded randomness, and grades two of its prompts wrongly.
• Gemma's 15.7-point lead over Qwen clears the noise easily. A 2-point lead wouldn't have.

Gemma's lead carries about ±4 points of error

Gemma got 87.6% of the prompts right and Qwen 71.9%, a gap of 15.7 points. The 95% confidence interval on that gap runs from 11.7 to 19.7 points, about ±4. The rule fixed in advance was to call the models different if that interval excluded zero. It does, and the second pass gave 16.8 and the same answer.

Comparing the models prompt by prompt keeps the error bar as tight as it is. Some prompts are hard for both models, and comparing on the same prompts cancels that out. Treating the two scores as unrelated would have made the interval 18% wider. Grouping instructions that share a prompt, as Miller recommends, barely mattered: each model's instruction-level error bar widened by 0.8% for Qwen and 3.9% for Gemma.

About 78% of that error's variance is the luck of the prompt draw

The interval narrows roughly with the square root of the number of prompts: ±8.7 points after 100 prompts, ±6.6 after 200, and ±4.0 at all 541. In IFEval's own order it cleared zero after 15 prompts, and every interval from 10 prompts on covered the final 15.7.

That last part is partly the luck of the order. Of the chart's first 200 shuffled orders, 39 left out the final 15.7 at some point after 50 prompts, and 18 after 100. Watching an interval and stopping the moment it looks decisive makes misses like that far more likely, which is why the test planned in advance reads the interval once, after all 541 prompts.

The ±4 points mostly reflects which 541 prompts IFEval happens to contain. In Miller's terms, the benchmark is one draw from a larger pool of similar prompts, and a different set of 541 would give a different gap. Only more prompts narrow that down.

At IFEval's size, the smallest gap this test would detect 4 times out of 5 is 5.7 points for this pair, and about 6 for two models closer together. Detecting a 2-point gap would take about 4,800 prompts, nine times as many as IFEval has. Miller's power analysis, using illustrative numbers, suggests new evals have at least 1,000 questions.

Asking each prompt more than once helps much less. In the sampling runs, answering each prompt 8 times instead of once narrowed each model's error bar by an estimated 13 to 14%, and past 4 answers per prompt each extra answer narrowed it by less than 1%. That is the diminishing return the paper predicts once the variance between prompts dominates.

Running the same eval twice gave different results

Each run with randomness off sent the prompts through twice at each load, with nothing changed in between: the same prompts, in the same order, through the same server. With randomness off the model always picks its most likely next word, so the second kind of chance from the opening should disappear and the passes should match exactly. One request at a time, on a 200-prompt subset, Qwen's did. With 8, 32 or 64 requests at a time, most responses came out different somewhere, and 4 to 7% of Qwen's prompts flipped between right and wrong. At 32 requests at a time, 39 of Qwen's 541 prompts flipped, and 13 of Gemma's.

For Qwen, the cause is batching. vLLM serves many requests quickly by running them through the GPU together, and the arithmetic for one request then depends slightly on which other requests share its batch. When two candidate next words are nearly tied, that tiny difference can tip the choice, and the response goes a different way from there.

The flips went both ways, with no detectable lean toward right or wrong, and scores moved by 0.2 to 1.5 points between passes.

Gemma's results flipped less often than Qwen's, but the two weren't batched the same way. Gemma's server preempted requests about 860 times per pass, evicting them and recomputing them later, which vLLM does when it runs out of KV-cache memory. Qwen's never did, so the difference doesn't show that Gemma is steadier. Part 3 has every comparison.

Averaging out rerun noise would shrink the error bar only from ±4.0 to ±3.5

The rerun noise is already inside the error bar, since every pass had it. It makes up 21.7% of the variance of Gemma's lead. Width goes with the square root of variance, so that fifth of the variance is only about an eighth of the width. Averaging many passes of each prompt would remove it, narrowing the error bar on the lead from ±4.0 points to ±3.5, and on Qwen's score from ±3.8 to ±3.4.

A deterministic server wouldn't narrow it even that much. Every rerun would repeat the same answer, but each prompt would still get only one answer, landing on one side of a near-tie or the other, and that varies from prompt to prompt like any other difficulty. Miller makes the same point about lowering the sampling temperature to cut variance: it moves the variance into each question's expected score, where resampling can no longer remove it.

The noise matters more for the gap than for either score. Comparing prompt by prompt cancels out how hard each prompt is, which both models share. It can't cancel rerun noise, because each model's pass has its own.

With batch-invariant mode requested, reruns matched exactly

vLLM has a setting, VLLM_BATCH_INVARIANT=1, meant to make each request's result independent of its batch. With it requested, Qwen's two passes at 64 requests at a time matched on all 541 prompts, against 78 of 541 without it. Throughput was 2% lower.

vLLM logged nothing confirming the mode was on, and the rule fixed before the run also asked for corroboration: the slowdown the deterministic code was expected to cause, more than 10%. There wasn't one, so by that rule the result stays unconfirmed. In hindsight, a speed comparison on two passes per side was too weak to decide it (part 4).

IFEval's official grader uses unseeded randomness

IFEval's reference grading code uses randomness in two places and never fixes it. One is a language detector that samples the text at random, which 95 of the 834 instructions rely on. The other turned up when the grader was tested: two prompts ask for a count of # or !, and the grader only accepts letters, so it silently counts a random letter instead. Grading the same responses twice gave different results until both were pinned with fixed seeds.

Pinning made grading repeatable, but those two prompts are still graded wrongly, because they're now checked for the letter y. They stay in, as published, so the scores stay comparable with other IFEval results (part 5).

The plan was fixed before any run, and every run checked itself

• The plan was code. The one yes-or-no question, how it would be computed and decided, and which runs would count were committed as code before any run. Only two changes came after the first runs had started. One changed two server settings for both models, before any counted run; the other filled in parts of the analysis. Neither changed how the comparison is computed or decided, and both are listed with when and why (part 1).
• Every run checked itself. Before its numbers could be used, each run confirmed it was submitted with the planned settings and loaded the planned weights, kept the planned number of requests going, recorded an answer for every prompt, and ran a self-test of the grader. None of the checks looks at accuracy, so no run could be thrown out for an inconvenient score. All six counted runs cleared them.
• Every run had its own server, so no one else's requests shared its batches.
• Every measured number traces to one results file, to the run records, or to per-prompt results exported from the same runs, and the analysis code gave byte-identical output twice.

What this doesn't show

• Which model is better in general. The gap mixes size, maker, training, compression and chat format, and it comes from one benchmark.
• Anything about tasks unlike IFEval's. Its prompts aren't a random sample of real requests.
• How much rerun noise other GPUs, vLLM versions or server settings produce. It was measured on one setup.
• How much results change between runs, each on a freshly started server. Every rerun here used the same server.
• How much Gemma's results change at other loads. Most rerun measurements used Qwen.

Part 7 has the full list.

The details

1. 01DesignHow the study was set upYes-or-no questions: 1
2. 02FindingPairing narrowed the error bar, and more answers per prompt stopped paying after fourStandard error of the gap: 2.02 points
3. 03FindingA batched rerun flipped 2 to 7% of prompts, with no detectable leanResponses identical, batched: 13.5–33.6%
4. 04InconclusiveWith batch-invariant mode requested, reruns matched exactlyIdentical responses, setting requested: 541 / 541
5. 05Problem hitIFEval's official grader uses unseeded randomnessInstructions using random detection: 95 of 834
6. 06Problem hitWhat went wrong during the runsRuns counted: 6 of 8
7. 07LimitsWhat this study doesn't showSmallest detectable gap: 5.7 points