Ai2 Replaces Priority Scheduler with Budget-Based GPU Allocation

Managing GPU clusters at scale means balancing availability, occupancy, impact, and utilization — a pyramid where each layer depends on the one below. At Ai2, the AI Infrastructure team oversees thousands of NVIDIA H100, B200, and B300 GPUs across clusters of 88 to 1,024 GPUs, serving roughly 150 researchers running everything from LLM and VLM training to robotics RL simulation and scientific agentic post-training. Demand consistently outstrips supply by 2–3×, turning every GPU hour into a contested resource.

Why the old priority model failed

The previous priority-based scheduler allowed teams to mark workloads as non-preemptible, leading to predictable pathologies: researchers parked no-op jobs to "squat" GPUs for future debugging, priority inflation pushed 100% of workloads to HIGH, and on-call engineers wasted hours negotiating shutdowns for maintenance. Lower-priority work starved entirely, and the system rewarded gaming over genuine need.

The new approach replaces ad-hoc priority negotiations with a transparent administrative budgeting process. Each project receives a GPU time budget; hierarchical fair-share allocation distributes unused capacity proportionally; and a time-slicing contract ensures preemptible workloads yield gracefully. The result: scheduling debates shift from operational firefighting to predictable, auditable resource planning.

Why it matters for GPU / AI infrastructure: As cluster sizes grow and workloads diversify, priority-based schedulers become bottlenecks. Budget-driven, fair-share models with explicit preemption contracts deliver higher impact per GPU hour, reduce operational toil, and give leadership a clear lever for aligning compute allocation with strategic priorities — critical for any organization running large-scale AI training on shared infrastructure.