Step 3 of 15Adopted

Skip vLLM's memory-profiling pass

Pinning the KV cache size skipped a profiling pass: engine start 110 s to 99.5 s, Ready 141 s to 131 s. A shortcut that left graph memory out of the KV budget ran out of GPU memory and was rejected.

Engine start
110 → 99.5 s
median of 3
Time to Ready
141 → 131 s
−7%
Shortcut arm
out of memory 2 of 3
Step 02 · Baseline
Production config
141 s
Step 03 · Skip the memory-profiling pass
Skip graph accounting
no clean run
Pin the KV cache
131 s
Both
130 s

Hover or tap a segment for its time. The twin's readiness probe runs every 10 s, so totals land on 10-second steps and the last segment absorbs the wait.

With the profiling pass gone, the rest of the engine start grows by about 40 seconds. The first-use kernel cost moved instead of disappearing.

Problem

Before it sizes the KV cache, vLLM runs a pass that measures how much memory the CUDA graphs will need. On the baseline that pass took about 52 seconds.

Change

Three arms. One pins the KV cache size with --kv-cache-memory, so the pass isn't needed. One sets an environment variable that leaves graph memory out of the KV-cache budget; vLLM still runs both captures. The third does both.

Result

Pinning the cache cut the engine start from 109.5–111.9 seconds to 99.2–99.6, with the same cache size. Ready fell from 141 to 131 seconds, about 7%, but that is a single step of the 10-second probe, and one of the three pinned runs still landed on 141. The profiling pass had also been paying first-use kernel costs. With it gone, those costs moved into the rest of the engine start instead of disappearing, so the saving was ten seconds, not fifty.

The environment-variable arm ran out of GPU memory in two of three runs on the shared card, with the graphs allocated on top of a cache sized without them. The run that survived came up with a third of the KV cache, which I didn't trace before rejecting the arm.

Decision

Keep the pinned cache size. At 7% it was under the 20% rule on its own; I kept it because every later arm built on it, and step 8 judged the combination. Reject the shortcut.