Share of prompts each model got right, randomness off, with 95% intervals
From the first pass of the main runs, 541 prompts each. The comparison itself is made prompt by prompt, which gives a narrower interval than these two side by side suggest.
The main comparison
| Value | |
|---|---|
| Gemma's lead, strict grading | 15.7 points |
| 95% interval, comparing prompt by prompt | 11.7 to 19.7 points |
| Prompts both models answered | 541 |
| Standard error, prompt by prompt | 2.024 points |
| Standard error, as two separate scores | 2.396 points |
| How closely the models' results track each other (correlation) | 0.302 |
| Rerun noise's part of the variance | 0.888 of 4.095 points², or 21.7% |
| Exact McNemar test, a check | p = 3.9 × 10⁻¹⁴ |
| Decision | different, because the interval excludes zero |
| Second pass, same prompts and server | 16.8 points, interval 12.9 to 20.8, same decision |
Why comparing prompt by prompt helps. This is the paired analysis Evan Miller's Adding Error Bars to Evals recommends wherever it's possible (section 4.2). Both models answered the same prompts, so a prompt that is easy or hard for both cancels out when you subtract one model's result from the other's. With a correlation of 0.302 between the two models' results, the standard error of two separate scores is 1.18 times the prompt-by-prompt one. That would have given an interval from 11.0 to 20.4 points.
How much rerun noise adds. Rerun noise that is independent from prompt to prompt is already inside the spread the interval is computed from. Its share of the variance is the rerun term planned in advance, (f_Qwen + f_Gemma) / (2 × 541), divided by the squared standard error, where f is each model's share of prompts that flipped between right and wrong from one pass to the next. That comes to 21.7%, so about a fifth of the variance of the gap is rerun noise. It is a bigger share than in either model's own score (11% for Gemma and 18% for Qwen at 32 requests at a time), because comparing prompt by prompt cancels out how hard each prompt is, and it can't cancel rerun noise, which each model's pass has separately (part 3).
What the second pass checks. The second pass used the same prompts on the same server. It checks the decision against rerun noise only, and says nothing about a different set of prompts.
Instructions graded together
IFEval puts one to three instructions in each prompt, and the instructions in one prompt share a response. An error bar that treats all 834 instructions as independent comes out too narrow whenever instructions in the same prompt are followed or missed together. A clustered error bar groups them by prompt to correct for that, as Miller's section 2.2 describes. The design effect is the clustered variance divided by the independent one, so its square root says how much wider the clustered error bar is.
| Qwen | Gemma | |
|---|---|---|
| Share of instructions followed | 79.9% | 91.2% |
| Standard error, instructions as independent | 1.390 points | 0.979 points |
| Standard error, grouped by prompt | 1.400 points | 1.018 points |
| √ design effect | 1.008 | 1.039 |
| Intraclass correlation, ICC(1) | −0.017 | −0.003 |
Grouping by prompt widened the error bar by 0.8% for Qwen and 3.9% for Gemma, the small correction predicted in advance, so it barely matters on this benchmark. The intraclass correlation, a second measure of how much instructions in one prompt move together, came out slightly negative for both models, close to zero. It comes from a different estimator (an analysis of variance), so the two needn't agree in sign, and both say grouping by prompt barely moves the error bar here.
Strict versus loose grading
The loose grader retries each instruction on eight versions of the response: as written, with markdown asterisks removed, with its first line, last line or both removed, and those three trimmed versions with asterisks removed too. An instruction counts as followed if any version follows it. So loose minus strict measures how often strict grading marks a prompt wrong only for formatting around an answer that otherwise complies.
| Qwen | Gemma | |
|---|---|---|
| Loose minus strict, with 95% interval | 2.2 points (1.3–3.8) | 2.8 points (1.7–4.5) |
| Prompts rescued | 12 of 541 | 15 of 541 |
| Instructions where the two gradings disagree | 15, of 7 kinds | 18, of 9 kinds |
| Most common kind | forbidden words, 6 | forbidden words and sentence counts, 4 each |
The results record which instructions the loose grader rescued, but not which of the eight versions rescued them, so I don't characterize either model's formatting further. The letter-count instruction with the grader bug from part 5 doesn't appear for either model, as predicted.
Several answers per prompt
Standard error of the score in points, against answers per prompt, with sampling randomness on
Move across the chart for exact values. Only 8 answers per prompt was measured (the filled points). The curves follow from splitting that data into variation between prompts and variation between answers to the same prompt. The dashed lines are the between-prompt floor: more answers per prompt never go below it, and only more prompts lower it.
The sampling runs had each model answer every prompt 8 times with randomness on (temperature 0.7). For each prompt, the share of its 8 answers that strict grading marked right varies for two reasons: some prompts are harder than others, and the same prompt comes out right on some answers and wrong on others. More answers per prompt only shrink the second part, which is why Miller's section 3.1 predicts quickly diminishing returns once the first part dominates.
| Qwen | Gemma | |
|---|---|---|
| Variance between prompts | 0.1451 | 0.0844 |
| Variance within a prompt | 0.0635 | 0.0332 |
| Standard error with 8 answers, measured | 1.682 points | 1.279 points |
| Standard error with 1 answer, estimated | 1.964 points | 1.474 points |
| Answers past which one more cuts the error by less than 1% | 4 | 4 |
| Prompts used / dropped for a failed request | 541 / 0 | 541 / 0 |
Going from 1 answer to 8 cut Qwen's standard error by 14.4% and Gemma's by 13.2%. For both models the variance between prompts is the larger part, and only more prompts reduce it.
With sampling on, Gemma's lead averaged over 8 answers was 16.0 points (standard error 1.55). That measures something different from the main comparison, which has randomness off, so the two numbers aren't compared or subtracted.
How many prompts the next eval needs
Smallest gap in points the test detects 4 times in 5, by number of prompts
Move across the chart for exact values. At 95% confidence and 80% power, holding the spread of this pair's prompt-by-prompt differences. Two models that disagree as often but sit closer together have a slightly larger spread: 6.0 points at 541 prompts, 772 prompts for 5 points and 4,824 for 2.
This is the power analysis of Miller's section 5. The spread of the prompt-by-prompt differences in the main comparison is a standard deviation of 0.471. With that spread, 95% confidence and 80% power, the smallest gap this design detects at 541 prompts is 5.7 points.
That holds the spread fixed at this pair's value, and the spread depends on how often two models disagree and on how far apart they are. For two models that disagree on as many prompts as these do (133, or 24.6%) but have a small true gap, the spread is slightly larger. The smallest detectable gap then rises to about 6.0 points, and detecting 5 points would take about 770 prompts, and 2 points about 4,800.