GPU Servers for AI: Hosting, Isolation and Production Readiness

Start below the operating system

The client’s document platform inventories the host and facility before promising that a device is ready for its pilot.

Real cloud examples make the boundary concrete. AWS currently documents p6-b200.48xlarge and p6-b300.48xlarge offerings with eight GPUs each. Its product table reports 1,432 GB and 2,144 GB of GPU memory respectively, alongside different host-memory and network capacities. These diagrams preserve that table’s scope; they are not live tenant telemetry or a claim that I operate these instances.

Do not substitute a DGX reference-system bill of materials for the cloud instance. NVIDIA’s DGX B200 guide lists 1,440 GB across eight GPUs, two fifth-generation NVSwitch devices and ConnectX-7 networking. Its DGX B300 guide lists eight 288 GB GPUs and ConnectX-8 networking. Provider-exposed memory, model allocations and reference-system specifications can differ; inspect the actual ordered configuration rather than silently reconciling unlike numbers.

A server combines CPUs, system RAM, GPUs, storage controllers, network interfaces, firmware and power supplies. PCIe and NUMA topology influence which CPU, memory bank and devices exchange data efficiently. A rack adds power distribution, cooling and network uplinks. These dependencies remain present when a cloud API hides them.

Verify rack power density, cooling compatibility, circuit redundancy, link capacity and the ownership of firmware updates before promising usable GPU capacity. Nameplate power is a planning limit, not measured energy. The facility must support the actual equipment and failover behavior, including maintenance conditions.

Know what the hypervisor isolates

The next boundary is the machine the application believes it owns. Hypervisors and containers divide responsibilities in different ways.

A hypervisor presents virtual CPUs, memory and devices to guests and controls access to underlying resources. Different platforms implement this differently: KVM uses Linux virtualization support, while cloud providers may combine a hypervisor with specialized offload hardware. Bare-metal hosting exposes a different responsibility boundary.

A VM has its own guest kernel. Ordinary containers share the kernel of their host, which may itself be a VM. Namespaces, cgroups and container policies are useful controls but are not automatically equivalent to a hardware virtualization boundary. Choose tenancy and threat assumptions before selecting the packaging format.

GPU passthrough, MIG and sharing are not synonyms

Sharing the accelerator sounds economical until isolation and contention enter the discussion. You now distinguish passthrough, MIG and time slicing.

With device passthrough, a guest uses an assigned device through a supported driver and virtualization path. IOMMU protection and device isolation groups matter for DMA; peer devices, firmware and platform support can affect the boundary. Do not treat an unsafe passthrough workaround as tenant isolation.

On supported NVIDIA GPUs, MIG partitions particular compute and memory resources into supported instance profiles. It does not partition the entire server or remove host administration responsibilities. Time slicing schedules access to shared hardware; advertising more logical GPU allocations does not create more VRAM or equivalent hardware isolation.

Separate scheduling capacity from serving capacity

The scheduler can count devices, but the service needs complete, usable replicas. The placement example shows why those are different answers.

A model replica using tensor parallelism may require several compatible GPUs together. For eight GPUs, a two-GPU reserve and two GPUs per replica, the arithmetic ceiling is floor((8 − 2) / 2) = three replicas. If the devices are in incompatible hosts or links, even that ceiling may not be achievable.

Check GPU memory per device, CPU and RAM for preprocessing, network bandwidth, storage access and placement constraints. A scheduler allocation is not a loaded model, and a loaded model is not proof of healthy latency under traffic. Readiness should reflect the state required to serve useful requests.

Storage and CPU can dominate cold starts

The team traces cold-start work because its preparation-time model depends on making the same artifacts ready sooner.

Model startup may include image retrieval, weight download, checksum verification, deserialization, device transfer, compilation and warmup. Separate those intervals. An ideal 40 GB transfer over a sustained 2 GB/s path already needs 20 seconds before other work; that is a lower bound with decimal units, not a startup measurement.

For an active model cache, fast local NVMe can remove repeated network retrieval when lifecycle and consistency rules allow it. It is not durable model ownership: keep versioned artifacts and recovery sources. Track cache misses, storage throughput, CPU bottlenecks and per-stage startup time instead of assuming every slow start is a GPU problem.

Price facility energy without double counting

The conversation reaches the facility bill. You keep measured energy, planning assumptions and bundled colocation charges from becoming the same number.

Power Usage Effectiveness compares total facility energy with IT equipment energy over a defined boundary and period. A planning estimate with 5 kW of average IT load and PUE 1.4 implies 7 kW including facility overhead. Over a 730-hour month at $0.15/kWh, that is $766.50. Real metering, tariffs and load shape determine the actual bill.

Do not multiply a colocation quote by PUE if its power charge already includes the same overhead. Identify whether billing covers reserved circuit capacity, metered energy, a rack bundle or another basis. PUE does not measure useful model work or total environmental impact; neither a lower PUE nor a lower GPU wattage alone proves a better service.

Design around the failure domain you can lose

A reserve looks reassuring until the whole host disappears. The recovery story must survive the actual failure domain, not merely a smaller allocation.

Two spare GPUs in the same eight-GPU host cannot recover the service if that host fails. A host, rack, switch, power circuit and availability zone are different failure domains. Place surviving capacity and data where the intended failure cannot remove them together.

Exercise a drain, an abrupt node loss, a missing model artifact, a slow storage dependency and a rollout overlap. Measure time to healthy capacity and the client-visible error or queueing interval. Backups need restore tests; redundant power supplies need genuinely independent upstream power to provide the intended protection.

Make production acceptance observable

The platform review closes only when readiness and recovery—not a server specification—support the customer pilot.

Track user-facing latency, errors, queue age and useful throughput alongside GPU memory, thermal or power throttling, device errors, CPU pressure and storage or network saturation. Correlate application traces with the particular model version, replica and host. GPU utilization is diagnostic context, not the service-level objective.

Document the owners of firmware, drivers, hypervisor, guest images, credentials, model artifacts and incident response. Gate releases on compatible versions and a reproducible rollback path. Hosting selection becomes much clearer once these responsibilities and recovery requirements are priced explicitly.