Eight short jobs submitted together, through the one-job lock
submitted, waiting. running, from the job's own start stamp to its end. Each job takes about 5 s from start to finish, a fraction of a second of it on the GPU: a 4096×4096 matrix multiply and a tiny language model, both checked against a CPU reference. Most of the batch's 2.7 minutes went to hand-offs between jobs.
Problem
Every GPU job's workflow step took an Argo semaphore with room for one, the lock, so jobs ran strictly in turn. Nobody had measured what that costs when many small jobs arrive together: how fast the queue drains, how much each job pays in overhead, and what a failure does to the jobs behind it. Without that baseline, a number for jobs at once has nothing to compare against.
Change
None to the platform. Two batches of eight jobs, each job its own submission, fired together: the first doing only its five seconds of work, the second also holding the GPU idle for 30 seconds, so overhead and work could be told apart. In the second batch, the third job to start was cancelled mid-run. Before each batch I checked that the CI queue was idle and no GPU job was running.
The gap between one job ending and the next starting, by part
Hover or tap a part. The middle three parts come from pod and queue timestamps read from the cluster during the run, at one-second resolution, not from the run record. The last runs from the container's start to the job's own start stamp, and the first is the gap less the other four, so it also carries their rounding. Most of every gap passed before the next job's pod existed, while Argo released the one-job lock and started the next waiting workflow's step.
Result
The short batch drained in 2.7 minutes, 2.96 jobs a minute; the 30-second batch took 6.7. No two jobs overlapped, and every output matched its CPU reference. Each job ran for about five seconds plus its hold, and the gaps between jobs were 16.1 seconds at the median in the short batch and 18.1 in the longer one. The queue admitted every pod in the same second it was created. Most of each gap came before that: 8 to 11 seconds in the short batch, and 10 to 12 in the longer one, from one job's container exiting to the next job's pod existing, while Argo released the lock and started the next waiting workflow's step. These runs don't isolate why that takes Argo 8 seconds or more. Container start took 2 to 3 seconds, and the job's own start-up to its start stamp, Python, PyTorch, CUDA and part of opening its run record, another 3 to 4.
The cancelled job closed 5.3 seconds after the request, as cancelled, and was
not retried. The next job started 15.7 seconds later, the shortest gap in its
batch, and all five jobs queued behind the cancelled one finished.
Decision
This is the baseline. One at a time, a short job waits about three times its own run time for hand-offs. With the lock raised, jobs up to the new limit start together instead of in turn (step 5). Container and process start-up are per job and remain.