Step 7 of 15Adopted

Persist the CUDA driver's kernel cache

Keeping the driver's compiled-kernel cache cut the start from 130 s to 51 s with graphs and from 80 s to 40 s without. The largest single gain.

Ready, no graphs
80 → 40 s
Ready, with graphs
130 → 51 s
First request, no graphs
44.7 → 5.1 s
Cache size
69 MB
Step 06 · Framework kernel caches kept
All graphs, caches kept
135 s
No graphs, caches kept
125 s
Step 07 · Driver kernel cache kept
All graphs, driver cache
56 s
No graphs, driver cache
45 s
All graphs, autotune off
135 s
No graphs, autotune off
125 s

Hover or tap a segment for its time. The twin's readiness probe runs every 10 s, so totals land on 10-second steps and the last segment absorbs the wait.

With the driver's cache kept, the 45 seconds of engine start and the 44-second first request both disappear.

Hypothesis

On this GPU generation, the driver compiles part of the image's kernels from PTX the first time they run, and caches the result in ~/.nv/ComputeCache. That directory disappeared with every pod.

Change

CUDA_CACHE_PATH points at the persistent volume. The cache is 69 MB.

Result

With CUDA graphs, the start fell from 130 to 50–51 seconds, graph capture from 52 to 12–13, and wake to first answer from about 135 to about 56. Without graphs, the start fell from 80 to 40 seconds and the first request from 44.7 to 5.1. A 23-second FlashInfer autotune step at every start was the same compilation: skipping autotune on its own changed nothing.

Only the cache location changed between the arms. The run that filled the cache paid the full cost, the runs after it didn't, and production repeated the pattern (step 8), so the cache is the cause. Which kernels the driver compiles is inferred; no driver-level trace confirmed it. A driver upgrade will make the next start pay the compile once and refill the cache.

Decision

Adopt the cache. With it warm, graphs cost about ten seconds more start and save almost nothing on a short answer, which reopened graphs-or-not for production.