Part 3 showed that batching makes identical passes disagree. vLLM has a setting meant to stop that, so one run tested it, with a rule for reading the result fixed in advance.
What the setting is
vLLM's batch-invariant mode, switched on with VLLM_BATCH_INVARIANT=1, uses
GPU code whose result for a request doesn't depend on what else is in the
batch. The batch-invariant run sent Qwen's prompts through twice at 64
requests at a time with the setting requested, to see whether the rerun noise
goes away.
There was a reason to doubt the setting would take effect. It might not support this GPU generation, and an unsupported setup could fall back to the ordinary code without saying so. So every result from this run carries a label fixed in advance: batch-invariant mode requested; activation unconfirmed by any engine-side signal. If the server had refused the setting, failed to start or failed every request, the run would have counted as untested.
What happened
- Matching. The two passes gave identical responses on all 541 prompts (95% interval 99.3–100%). The same load without the setting matched on 78 of 541 (14.4%, interval 11.7–17.6%). The two intervals are nowhere near each other.
- Speed. The average tokens per second of the two passes with the setting, divided by the same for the two passes without it, is 0.979.
- Batching. The passes really were batched: 47.9 and 49.7 requests running on average, against 51.0 and 49.8 without the setting, and no preemptions in either.
- vLLM's own signal. A scan of vLLM's startup log found no line confirming the mode. That says nothing either way: vLLM's source history, as of the change that made it refuse the mode on unsupported GPUs, shows no line that confirms the mode on a supported one. So the scan was never part of the self-checks or the reading rule.
The rule fixed before the run
With no confirmation from vLLM, the rule looked for two signals: perfect matching, and the slowdown the deterministic code was expected to cause. The analysis code applies it mechanically.
- "Working" requires the lower end of the matching interval to be at least 0.99, and throughput below 0.9 times the run without the setting. Matching met it, at 0.993. Throughput didn't, at 0.979.
- "Doubtful", meaning the run behaved like ordinary passes, requires the two matching intervals to overlap and throughput of at least 0.95 times. Throughput met it. The intervals don't overlap.
- Anything else is reported as observed, with no conclusion. That is where this run landed.
The matching threshold amounts to perfect matching at 541 prompts, and it was set that strictly on purpose.
Why the speed signal couldn't decide it
Throughput was 2.1% lower with the setting requested, far short of the 10% the rule expected. But two passes per side can't separate a difference that small from ordinary pass-to-pass variation. In this same study, the two passes of Qwen's main run differed by 32% in wall time, about 177 and 134 seconds by the once-a-second sampler. The first pass with the setting requested was also the first work its server did, while the two passes without it were the fifth and sixth on theirs. So the 2.1% isn't a measured cost of the setting.
The rule assumed a slowdown that didn't come
The rule assumed a working batch-invariant mode would visibly slow the server. It barely did. Either the mode is cheaper on this GPU than assumed, or something other than the mode made the passes match, and neither the rule nor these two passes can tell those apart.
The speed half of the rule was also weaker than it allowed for. It compared two passes against two, at different points in their servers' lives, with no estimate of how much speed varies between passes. A next version should take that variation from the repeat passes the study already has.