The general GPU queue's quota, with no small job fitting
Up, the 7B borrows one share and 12,200 MiB through the large-model queue, which leaves 404 MiB of GPU memory. A small job declares 1,024, so no lowest-priority job is admitted until the 7B is scaled back to zero. A job at training priority would evict the 7B instead.
Problem
Midway through the batches in step 5, a request through the lab's LLM gateway woke the on-demand 7B. Two jobs submitted 27 seconds later, at the lab's lowest GPU priority like the rest of step 5's batches, didn't start for 30 minutes.
Result
Up, the 7B runs through the large-model queue, which borrows one GPU share and 12,200 MiB of GPU memory from the general GPU queue. That leaves 404 MiB, and each small job declares 1,024, so the queue admitted neither job; their pods existed but were held at the queue's gate. The lab's other on-demand model, a Gemma 12B, declares the same 12,200 MiB, so it would hold them off too.
Two lowest-priority jobs submitted while the on-demand 7B was up
submitted, waiting. running, from the job's own start stamp to its end. Both jobs waited in the queue for 30 minutes without failing. Once I scaled the 7B to zero by hand, the queue admitted both within 7 s, and they ran normally.
The jobs didn't fail: the job time limit counts from when a pod starts, and a held pod never starts. The 7B's idle reaper scales it down an hour after its last request, and rather than wait the rest of that hour I scaled it to zero by hand. The queue admitted both jobs within seven seconds, and they ran normally.
Priority changes the outcome. A job at training priority, a GPU step's default, reclaims the borrowed share and evicts the 7B, which took four seconds against a test copy of the 7B in the cold-start series. In the other direction, the queues are configured to let a waking 7B evict running lowest-priority jobs, which Argo then retries. That direction wasn't tested.
Decision
No change yet. Lowest-priority jobs that wait don't fail, but each request keeps the 7B up for another hour, so steady traffic holds them off indefinitely. A run that needs the card can claim the lab's GPU window, an exclusive claim that clears the on-demand models within one five-minute tick and holds off their wakes; these test jobs didn't claim one.