Training a large model and serving it to users are different jobs, and they need different facilities. Get the match wrong and you either overpay for infrastructure you don't need, or you sign a lease on a hall that can't support the hardware.
What inference and fine-tuning actually ask of a site
Inference is the job of running a trained model in response to user requests. Fine-tuning is adjusting an existing model on a smaller dataset. Both workloads typically run on 8-GPU servers, and the standard versions of those servers are air cooled.
An H100 or H200 server draws about 10 kW. A B200 server draws about 14 kW. Those are per-server figures. If you put 10 servers in a hall, plan for 100 to 140 kW of power drawn by the servers themselves, plus the facility's own overhead for cooling and power distribution. That kind of density, roughly 10 to 14 kW per rack, is within the range of a well-specified existing data center.
That matters because colocation, which is space and power for hardware you own, is easier to source when your density requirements fit what operators have already built. A hall designed for enterprise IT is often built to lower density than a modern AI-focused facility. A hall built for AI workloads can handle higher densities than a legacy enterprise hall, though the exact ceiling varies by site. Either way, the cooling approach is conventional: large air-conditioning units that circulate cool air through the hall, with no water pipes needed at the rack itself.
Inference also tends to scale horizontally. You add servers as traffic grows. That means you can start with a modest footprint and expand, which gives you more flexibility in how you commit.
What large-model training asks of a site
Training a frontier model from scratch, or running a very large pre-training run, is a different problem. The hardware of choice is rack-scale: systems like the GB300 NVL72, which holds 72 GPUs and 36 CPUs in a single rack and draws up to 142 kW per rack. Eight of those racks grouped together as a single deployable block, what NVIDIA calls a scalable unit, draw about 1.2 MW in total, including the networking equipment inside the block.
That density requires liquid cooling at the rack. In the GB300 NVL72, the GPUs and CPUs are on liquid cooling, and the other components in the rack are air cooled. The facility must supply a water cooling loop that reaches each rack. That is a meaningful infrastructure requirement: the site needs pumping equipment, the pipework fittings that distribute coolant to each server, and the plumbing to connect them, none of which a standard air-cooled hall has.
NVIDIA's own reference architecture for the GB300 system requires the data center to meet Uptime Institute Tier 3 design standards, which means the facility can be maintained and repaired while it stays live, and has no single point of failure. Not every colocation site qualifies.
Training runs also need low-latency, high-bandwidth connections between every GPU in the cluster. That means the GPUs must be physically close together. A training cluster is not something you spread across two buildings or two sites. The whole cluster needs to sit in one contiguous space and power allocation.
What this means for site choice
For inference and fine-tuning, the site requirements are less demanding. You need reliable power, adequate cooling density for air-cooled servers and enough network connectivity to reach your users. A Tier III facility covers the reliability bar. You have more sites to choose from, which affects how quickly you can get started.
For large-model training, the site must have liquid cooling infrastructure, high power density per rack, Tier III or better reliability, and enough contiguous capacity to hold the whole cluster. That is a shorter list of qualifying sites. Lead times to get the facility ready are longer, and the commitment you make is longer to match.
On term: whether you are contracting colocation or reserving GPU compute capacity from a specialist cloud provider that rents GPU time rather than general-purpose cloud services, the commitments in this market run 36 to 60 months. That is true for inference workloads and for training workloads alike. The difference is that a training cluster represents a larger total commitment, so the site-selection decision carries more weight.
Questions to take into a call with an operator
- What is the maximum kW per rack this hall supports today, without a facility upgrade?
- Is liquid cooling available at the rack, and what does the facility water loop look like?
- Does this site meet Tier 3 design standards, and is that documented?
- What is the largest contiguous block of space and power you can deliver in a single allocation?
- What is the lead time from contract to powered-on hardware?
- How does the contract handle expansion if my cluster needs to grow?
Sources
Power figures for H100, H200 and B200 8-GPU servers are planning figures in general use. GB300 NVL72 rack power of 142 kW maximum and the 1.2 MW scalable-unit figure are from NVIDIA's GB300 NVL72 reference architecture. The Tier 3 facility requirement is from NVIDIA's DGX SuperPOD reference architecture for DGX GB300. Uptime Institute Tier classifications are defined in the Uptime Institute Tier Standard: Topology.
Share your requirements. We'll canvas the market.
Tell us what you need (for colocation space and power: market, kW or MW, cooling and timing; for reserved GPU capacity: region, GPU type, cluster size and timing) and we'll canvas colocation and neocloud operators in parallel. Shortlist in 48 hours. We don't sell hardware. We broker colocation space and power, and reserved GPU compute capacity.
Earlier conversations get better terms. When you engage early, we have time to negotiate with vendors before you need to commit. We're paid only when a deal closes, and the fee structure is agreed in writing before any introduction.