Step 15 of 15Adopted

Fork the engine process

Forking the engine process brought Ready to 35–36 s. A cold caller's first token: 36.5 s median, with no 503s.

Production time to Ready
40 → 35–36 s
Cold caller, median
36.5 s

From the request to the first token, seconds

Gateway holds the request
41.4 s
Held, engine forked
36.5 s

the model reports Ready. a 503 sent to the caller. the first token. The same hold, on a model that starts about five seconds sooner because its engine process no longer re-imports every library.

The held request, before and after the fork.

Problem

Step 11 showed the engine process importing every library a second time, because it was spawned.

Change

VLLM_WORKER_MULTIPROC_METHOD=fork. At about 14% of the start it fell short of the 20% rule. I adopted it because decode didn't change and the usual risk doesn't apply: forking is unsafe once the parent has initialized CUDA, vLLM's API server never does, and vLLM falls back to spawn if it has.

Result

On the real pod, the time before the engine starts fell from 22 seconds to 17–18, and Ready from 40 to 35–36. A cold caller's first token came at 36.54, 36.50 and 41.48 seconds, with no 503s. The third run fell on the readiness probe's next five-second step.

From scale-up to Ready, by phase

Before: the twin 141 s
After: production 36 s

Hover or tap a segment for its time. The two rows come from different systems: the twin's readiness probe runs every 10 s and production's every 5 s, so only the engine phases compare directly.

Where a start's seconds went at the baseline, and where they go now. The 12 seconds of API server and readiness, a third of today's start, aren't attributed yet.

Decision

Adopted.

While verifying it I found an automated client, probably an uptime check, requesting the gateway's root and model list about every 30 minutes, and each request woke the 7B. With a 60-minute idle window, that can keep the model up indefinitely. Only inference requests should wake it, and that change is next.