Distributed training jobs possess a strict atomic scheduling requirement: either all N worker pods start simultaneously, or none should start. Partial scheduling causes deadlocks where multiple jobs hold parts of the cluster while waiting indefinitely for missing ranks.
The Distributed Deadlock Problem
Job A acquires 40 of 64 nodes; Job B acquires 24 of 64 nodes. Neither job can start execution because distributed collective communicators require all ranks to initialize. Both jobs hold GPUs idle forever without gang scheduling.
Kubernetes Kueue & Volcano Schedulers
Kueue and Volcano provide admission control and gang reservation in Kubernetes. Workloads wait in logical priority queues until the complete quota of GPU resources is available in a single atomic scheduling transaction.
Slurm Job Arrays & Heterogeneous Reservations
In HPC environments, Slurm manages exclusive node reservations with strict time limits, ensuring all ranks in an sbatch job allocation initialize concurrently.
MATHEMATICAL FORMULATION · CLUSTER STARVATION RISK MODEL
Without gang scheduling, as cluster utilization U increases toward capacity C_total, the probability of multiple competing distributed jobs entering an unresolvable partial allocation state approaches 1.0.
REAL-WORLD PRODUCTION ENGINEERING
Configure Kueue LocalQueues and ClusterQueues with strict queue preemption rules to allow high-priority pretraining jobs to preempt development runs.
Set pod-group minMember annotations in Volcano to match total distributed world size (world_size = nodes * 8).
Implement automated watchdog timers: terminate and requeue any job that fails to complete NCCL rank handshake within 120 seconds of pod creation.