Your GPU Cluster Has a Placement Problem, Not a Chip Problem

Buying another GPU is an easy infrastructure decision to explain. Explaining why expensive GPUs sit idle while a job waits is harder. An October 29 SkyPilot engineering post, now attracting attention on Hacker News, puts that uncomfortable question back on the table: how much of an AI cluster is actually usable for the workloads it was purchased to run? The answer depends on placement, failure recovery, and the shape of available capacity—not merely the number of working accelerators.

The Coe Lab’s view is that this is a procurement problem disguised as a scheduling problem. Hardware specifications describe individual components and their connections. A production service consumes combinations of those components over time. If the operating model cannot reliably assemble the right combination, nominal capacity becomes an accounting number rather than a useful promise to developers.

The news: capacity depends on where the GPUs are

In its new analysis, SkyPilot reports that, under a modeled 2% node-unavailability rate, 16-node slices offer 27% more usable capacity than 18-node slices on a 128-rack NVL72 cluster. It also describes rack-aware placement, spare nodes, and checkpoint-based defragmentation. These are vendor claims and modeling results, not a universal benchmark or a guarantee for another deployment. [Source: SkyPilot’s October 29 analysis.]

The distinction between modeled availability and delivered application throughput matters. More schedulable nodes do not automatically mean proportionally faster training. A workload could already be limited by storage, CPU preprocessing, memory pressure, or an application-level synchronization point. Treat the headline as a reason to inspect a cluster’s operating assumptions, not as a multiplier to paste into next year’s budget.

This topic had 137 points on Hacker News when selected for this article. That establishes current community interest, not technical validation. The useful discussion is broader than a particular scheduler: infrastructure teams need a vocabulary for explaining why apparently idle hardware can still be unavailable for a particular request.

Stop treating capacity as a single number

Consider an illustrative cluster with three racks and six free nodes in each. An inventory dashboard reports eighteen free nodes. A job requiring sixteen nodes inside one rack still cannot start. Nothing is broken, and no capacity has disappeared. The request and the inventory simply have incompatible shapes.

This is analogous to a hotel with eighteen empty single rooms and no available conference hall. Counting empty floor space will not tell an event organizer whether the building can host a meeting. Likewise, a GPU count cannot describe the availability of a required communication domain. The operational unit is not necessarily the device that appears on the purchase order.

A better capacity dashboard separates physical inventory, healthy inventory, placeable capacity, and productive output. Physical inventory answers what you own. Healthy inventory answers what passes checks. Placeable capacity answers what can satisfy an actual workload request. Productive output answers what useful work finishes. Each number can improve while another deteriorates.

That separation also changes incident triage. If healthy inventory falls, investigate hardware and maintenance. If health is steady but placement failures rise, inspect fragmentation and policy. If jobs start normally but useful output drops, examine application behavior and the data pipeline. Combining all three into an average utilization percentage makes the team slower precisely when a more specific diagnosis is needed.

Why topology belongs in the workload contract

NVIDIA’s rack-scale scheduling documentation describes NVLink domains and partitions as placement boundaries. It identifies cluster UUID and clique ID as topology identifiers that system software can use. This provides a concrete foundation for treating connectivity as part of a resource request rather than assuming every free GPU is interchangeable. [Source: NVIDIA’s rack-scale scheduling guide.]

The operational implication is that a request should describe both quantity and relationship. Asking for sixteen GPUs specifies a count. Asking for a tightly connected group specifies an arrangement. A platform that accepts only the first request may appear simpler while silently making the second decision on behalf of the application owner.

Not every workload deserves the strictest arrangement. Independent batch tasks may tolerate being scattered. A tightly coupled job may benefit enough from locality to justify waiting. The correct contract makes that choice visible: required placement, preferred placement, or unrestricted placement. Otherwise, the scheduler must guess whether starting immediately or preserving locality is more valuable.

Developers should supply evidence for those requirements. Benchmark representative work under different placements, record useful throughput, and compare total completion time including queueing. A job that runs faster after an hour of waiting may still finish later than a less-local placement that starts immediately. Conversely, accepting a poor placement can monopolize resources for longer and worsen everyone else’s queue.

This creates a practical opportunity for platform teams: offer documented workload classes rather than a menu of obscure hardware labels. A user should be able to select an independent batch class, a locality-sensitive distributed class, or a latency-sensitive serving class. The platform can translate those intentions into implementation details while retaining enough metadata for debugging.

Recovery headroom is not wasted capacity

NVIDIA’s Slurm guidance describes block scheduling and the --segment option for expressing atomic placement groups. It also discusses preferring sixteen-node segments over full eighteen-node segments so a domain with unavailable nodes can remain useful. The same guide covers IMEX integration for job-level GPU memory import/export isolation. Verify configuration against the deployed Slurm release before applying its examples. [Source: NVIDIA’s Slurm block-scheduling guide.]

The broader lesson is to price recovery headroom as insurance rather than classify it automatically as waste. A spare resource has value when it allows useful work to continue through a failure. Its value depends on the workload’s interruption cost, repair time, spare eligibility, and the probability that several resources become unavailable together.

Here is a deliberately simplified calculation, not a reproduction of a vendor benchmark. Suppose a hypothetical eighteen-node domain needs every node healthy to satisfy a request. If each node is independently available with probability 0.98, the probability that the entire domain is available is 0.98 raised to the eighteenth power, or roughly 69.5%. Strong individual availability can coexist with much weaker availability for a rigid group.

The independence assumption is the fragile part. Shared cooling, power, maintenance, or fabric incidents can affect multiple nodes at once. A calculation based only on unrelated component failures may overstate the protection offered by local spares. Teams should distinguish failures that a same-rack spare can absorb from failures that require another rack or a different facility.

Headroom also requires policy enforcement. If the scheduler fills every spare with ordinary work, recovery depends on reclaiming that capacity quickly and safely. Borrowing idle spares can be reasonable, but borrowed capacity must come with a clear interruption contract. Otherwise, an apparently efficient steady state creates a dispute between tenants during the next hardware fault.

Defragmentation is an application feature, too

It is tempting to treat defragmentation as an entirely internal scheduler operation. In reality, moving useful work has a cost that applications help determine. A task with validated checkpoints, bounded restart time, and reproducible inputs is easier to relocate than one that carries hours of undocumented state in memory.

Before enabling automatic relocation, define what survives interruption. Model state alone may not be sufficient. Depending on the application, the recovery contract can include optimizer state, random-number state, input position, scheduler state, and externally visible progress. The correct set is application-specific; a checkpoint file existing on disk does not prove that a run can resume correctly.

External side effects deserve particular attention. A resumed job might send a notification twice, write duplicate records, or repeat a paid request. Recovery testing should therefore examine correctness as well as runtime. Making operations idempotent, recording committed work, or separating computation from publication can reduce the consequences of a restart.

There is also a fairness question. If the platform repeatedly moves small workloads to make room for large ones, overall throughput may rise while one group’s experience deteriorates. Track preemptions, lost work, and completion-time distributions by tenant or workload class. A policy that looks excellent in aggregate can still be unacceptable for the people consistently paying its interruption cost.

A practical evaluation plan before buying more hardware

A useful evaluation starts with real demand rather than a synthetic full-cluster job. Collect the shapes and durations of submitted work, the placement constraints users actually need, and the reasons jobs wait. Preserve privacy by collecting resource metadata instead of prompts, training data, or sensitive workload contents.

• Inventory communication domains and confirm that scheduler labels match the actual hardware relationships.
• Record placement failures separately from ordinary capacity shortages and administrative holds.
• Compare a baseline policy with locality-aware placement using the same representative workload mix.
• Test reduced availability deliberately, including a local node failure and a broader domain outage.
• Validate checkpoints by resuming work and checking outputs, not merely by measuring checkpoint creation time.
• Measure tenant-level disruption before enabling automatic preemption or capacity borrowing.
• Define rollback criteria before changing the production scheduling policy.

The primary success measure should be useful completed work per unit of cost, accompanied by completion-time objectives. Include queueing time, retries, wasted computation, storage traffic, and operator intervention. A policy that improves a narrow kernel benchmark while producing more failed runs has not necessarily improved the service.

Run the comparison long enough to include churn. A freshly emptied cluster is unusually easy to schedule. The difficult condition is a shared cluster after varied jobs have started, finished, failed, and resumed. Replay a workload trace where practical, or stage a controlled trial that captures this changing state without disrupting critical production users.

For purchasing decisions, ask vendors to demonstrate degraded operation, not only a pristine launch. What happens when one node is unavailable? Which job sizes become impossible? How much capacity can be recovered without crossing the intended communication boundary? Those questions turn a throughput demonstration into a test of operational suitability.

What This Means for developers, IT leaders, and homelabbers

For developers, placement requirements should be explicit and justified. Avoid requesting the largest tightly coupled group merely because an example configuration does. Smaller or more flexible requests may start sooner and waste less scarce capacity, provided they still meet the application’s performance requirements. Keep the benchmark and the reason for the chosen shape alongside the deployment configuration.

For IT leaders, the actionable change is to buy an operating envelope rather than a maximum specification. Require evidence about workload mix, failure behavior, upgrade procedures, and recovery time. Assign ownership of topology metadata and scheduler policy. A team cannot be accountable for delivered capacity if another team changes physical relationships without updating the control plane.

For homelabbers, none of this implies that a small inference server needs enterprise orchestration. The transferable principle is to respect bottlenecks and recovery boundaries. Two GPUs in different machines do not automatically behave like a larger local accelerator. Benchmark the actual arrangement before adding distributed complexity, and keep a simple fallback for important services.

A homelab can also practice the economics of headroom at much smaller scale. Leaving enough memory for a restart, maintaining space for a second checkpoint, or reserving a modest CPU budget for recovery tooling may deliver more reliability than chasing maximum steady-state utilization. The right reserve is the one supported by a tested failure scenario, not an arbitrary percentage copied from a larger system.

Looking forward: demand evidence about schedulable service

The next useful infrastructure benchmark should report more than peak throughput. It should show which workload shapes remain available as components fail, how quickly fragmentation accumulates, and what users lose during recovery. Those measurements would make it easier to compare software policies and hardware platforms on terms that resemble production operation.

Our prediction is that workload contracts will become a more important differentiator than another layer of utilization dashboards. Platforms that can express locality, interruption tolerance, and recovery requirements clearly should be easier to operate than platforms that hide those choices behind a single GPU count. That is a direction to watch, not a claim that any one vendor has already solved it.

The immediate takeaway is straightforward: before asking how many more accelerators to buy, ask which current requests the cluster cannot satisfy and why. Sometimes the answer will genuinely be insufficient compute. Sometimes it will be the wrong placement policy, an excessively rigid application shape, or an untested recovery path. Knowing the difference is how expensive infrastructure becomes a dependable service.

SkyPilot’s October 29 analysis

NVIDIA’s rack-scale scheduling guide

NVIDIA’s Slurm block-scheduling guide

Hacker News discussion