Ai2 swapped GPU priorities for budgets: owed compute delivered 98% of the time

In a post on 9 October 2026, the AI Infrastructure team at Ai2 explains how it replaced a priority-based scheduler for its thousands of NVIDIA GPUs, shared by about 150 researchers. Under the old system, almost every job ended up marked high priority and people held GPUs with idle jobs. Now managers give each project a budget of GPU time, any job not covered by a budget can be paused for others, and each job declares a minimum run time during which it is protected. Over a 30-day test, teams received 98% of the GPU hours they were owed, while the clusters stayed 98% occupied. Wait times for short debugging jobs fell from 2 hours to 30 seconds at the 90th percentile, and repairs needing a human fell by 74%. Ai2 also lists what got worse: researchers' long interactive sessions now get interrupted, so it is building a separate CPU cluster for them.