Step 14 of 15Adopted

Hold the cold request

The gateway holds a cold request for up to 60 s until the model is Ready, then forwards it. A client that obeys the retry hint went from 150.3 s to 41.4 s, with no 503s.

Client obeying Retry-After
150.3 → 41.4 s
503s on a cold request
0
for starts under the 60 s hold

From the request to the first token, seconds

Client obeys Retry-After: 150
150.3 s
Client retries every 5 s
45.6 s
Gateway holds the request
41.4 s

the model reports Ready. a 503 sent to the caller. the first token. No 503 at all. The gateway keeps the request open, checks the model's health every second, and forwards it the moment the model is Ready. The caller sees one slow answer.

The held request never sees a 503, and it beats the polling client because the gateway checks health every second.

Problem

Answering a cold request with a 503 punished callers. A well-behaved client waited out a stale hint, and a client that doesn't retry 503s saw an error.

Change

The gateway holds a cold request for up to 60 seconds, checking the model's health every second with a bounded timeout on each check, then forwards it normally, streaming included. The model is reachable only once its pod passes the readiness probe, which runs every five seconds, so the request goes through within a second of Ready. Past the hold, Retry-After is 5 seconds. Sixty seconds is under the tightest known client timeout, a 120-second chat call.

Result

Over three runs, a client that obeys Retry-After went from 150.3 seconds to 41.4, with zero 503s. It also beat the five-second poller by about four seconds.

Decision

Adopted.