What went wrong during the runs

vLLM sets aside GPU memory for requests in progress (the KV cache). With the server limited to 76% of the card's memory and planning for requests up to 8,192 tokens long, it left Gemma 2.16 GiB for that. One 8,192-token request needs 2.63 GiB, so the server refused to start. Both figures come from the server's log. The run never reached its self-checks, and the analysis lists it as failed.

The fix was to plan for requests up to 4,096 tokens, for both models. IFEval's longest prompt is 1,858 characters, about 600 tokens, and responses were capped at 1,280 tokens, so 4,096 is twice what any request could need. No request's length limit changed. What did change is how the server packs requests into each step on the GPU. That packing can change results, which is what part 3 measures, so runs before and after the change aren't interchangeable. Both models moved together, so the comparison stayed fair.

Qwen's first main run had already cleared its self-checks under the old 8,192-token setting. Under the new plan it was a run with the wrong settings, so Qwen's main run was redone at 4,096. The analysis lists the first run as having the wrong context length and never reads it. A query of the study's runs lists eight: the six counted and these two.

Running one request at a time is slow, so the plan let the concurrency runs switch to a fixed set of 200 prompts (chosen with seed 0) for those passes if a full pass looked too long. A five-prompt probe, run one request at a time, predicts how long all 541 would take, and the switch happens above two hours.

All five probe requests succeeded. The model produced 19.23 tokens a second, and the probe's responses averaged 256.0 tokens. The prediction, 541 prompts times the average response divided by the speed, was recorded as 2.0005 hours, about two seconds over the line. The rule was applied as written. Both passes one request at a time ran on the 200 prompts, so every comparison involving them uses those 200, with intervals about 1.6 times wider.

In hindsight the probe was pessimistic. The two 200-prompt passes took about 1,210 seconds each by the server's once-a-second sampler, which puts a full pass nearer one hour than two. The sample counts are in the results, and that extrapolation is worked out from them here. Why the probe overestimated isn't established: it used only five prompts, and it was also the first work its server did.

Version 1 of the analysis code stopped on any run that didn't match the plan, and a run from before the change to 4,096 tokens now existed. It also left out the speed half of the batch-invariant rule and several counts the plan had promised.

Version 2 lists runs it can't use instead of stopping on them. It completes the reporting the plan already named: each model's score, counts of failed requests and cut-off responses, and the batch size and set-aside requests for each pass. It also fixed the rule for reading the batch-invariant run. It changed no formula, no part of the main comparison and no self-check. It went in after three runs had cleared their self-checks, with their numbers unread, and before the batch-invariant run existed or anyone ran the analysis. Version 1 was never run on the counted runs. Running version 2 twice gives byte-identical output.