Running GLM-5.1 on a rented 8× H200 node

In April I wanted a big open model behind my own coding agent, served by me, on hardware I could switch off. I picked GLM-5.1 in FP8 and rented one node with eight H200s from HPC-AI Cloud.

This page is about everything around the model: how I launched the node, kept it cheap, booted it quickly and watched it. The tuning results have their own write-up. The short version is one user, one stream, 22.25 tokens a second with the first token in under a second, at $9.44 an hour on spot.

One node is the right shape for one user

The plan in the repo starts from a point I still agree with. Eight H200s is vLLM's published deployment shape for GLM-5.1-FP8, and more GPUs mostly buy concurrency, not single-stream speed. I was the only user. A second node would've doubled the bill and left my one stream exactly as fast.

vLLM's own benchmark for this model on eight H200s reports 526.57 output tokens a second. That's 32 concurrent requests with 8k-token inputs, so it says nothing about how fast one person's answer streams. I estimated 30–45 tokens a second for a single stream with thinking off. I measured 22.25.

The region wasn't my choice. The provider's marketing page showed a cheaper US SKU, but the live control plane on my account only had the Singapore node in stock, so I ran it in Singapore and reached it from San Francisco.

Spot without surprises

Spot cost half as much: $9.44 an hour for the node against $18.80 on-demand, or $1.18 and $2.35 per GPU-hour when I set the repo up. The catch with this provider is that a spot-scheduled order can still be created at the on-demand unit price if the request payload is wrong. Nothing warns you. You find out from the bill.

So the control-plane CLI wraps every launch:

• It reads live catalog pricing before it creates anything, instead of trusting the prices written in the README.
• Right after launch, verify-billing --strict checks the order and any bill rows against the spot rate. A mismatch means stop, delete and recreate, because a mispriced live order can't be converted in place.
• Creation fails closed to spot. An on-demand launch needs an explicit opt-out flag.

Preemption means recreate, never repair. A stopped node with a preemption marker gets deleted and relaunched, and a failed start on a stopped spot node takes the same path. When the H200 spot market was empty, which did happen, the loop recorded waiting-for-spot-capacity and retried every 120 seconds instead of crashing.

Fast boots come from an image and warm caches

A cold node costs twice: you pay for eight idle GPUs while packages install, and you wait. I prepared one node by hand and saved a private image from it. The provider only accepts that while the source instance is stopped, and the node sits in SavingImage until the image is built.

The image carries the Python runtime and the DeepGEMM stack under /mnt/persist/glm51, so a new node doesn't reinstall either. With the shared volume attached, the Hugging Face weights and the Triton, TorchInductor and CUDA compiler caches live on it and survive instance deletion. Without it, the first boot on a brand-new node still downloads the model once onto local NVMe.

The bootstrap script prefetches the model into the hot cache before it starts vLLM. That keeps the slowest step out of the server's startup, where it's hard to tell a slow download from a hang.

The endpoint never faced the internet

vLLM listens on 127.0.0.1:8000 on the node. My Mac reaches it through an SSH tunnel on local port 18000, because 8000 was already taken locally, and the agent points at http://127.0.0.1:18000/v1. There's no public port to scan and no gateway to secure.

The tunnel script looks up the provider-assigned SSH port and refreshes a repo-local known-hosts file. A rebuilt node therefore doesn't mean clearing stale host keys by hand. The node can also join my tailnet, and when it isn't on it, the tooling falls back to the provider's public SSH endpoint.

Seeing it while the meter runs

A metrics script pushes a snapshot to a Pushgateway, Prometheus scrapes it, and a Grafana dashboard shows what I cared about while paying by the hour: hourly cost across running instances, account runway, spot stock for the H200 template, endpoint health and the latest benchmark results.

With spot, the stock panel mattered more than I expected. It told me whether a preempted node could come back at all, or whether I should stop trying for the day.

A tuning loop that lives off the GPU node

The optimizer loop runs in my home Kubernetes cluster, not on the spot node. The node is disposable and the loop's state shouldn't be. Each candidate is a set of environment flags for the vLLM bootstrap. The loop applies one candidate at a time, restarts vLLM only when the candidate changes, opens a temporary tunnel, and runs an endpoint probe, a single-stream benchmark and a reasoning prompt with thinking on.

Scoring is a plain weighted sum: output tokens a second times 10, reasoning tokens a second times 5, minus seconds to the first token, plus a Terminal-Bench term weighted at 1,000 that stayed empty in these runs. By default the loop keeps the incumbent and only tries something new when a planner proposes it, so a quiet pass leaves the server warm.

It compared two and three speculative tokens with MTP, and CUDA graphs on and off. Three tokens in eager mode won at 22.25 tokens a second. The post has the full table and the two passes that errored.

What one stream costs

At 22.25 tokens a second, one stream produces 80,100 tokens an hour. On spot at $9.44 an hour that's about $118 per million output tokens. On-demand at $18.80 it's about $235.

Running more streams at once is how that number comes down, and I didn't test it.

What I didn't measure

I'd store the speculative acceptance rate from the first run. vLLM logs it, and the repo has a script to tail those lines, but the loop never saved them, so I know which setting was faster and not how often its drafts were accepted.

I'd also benchmark more than one prompt before calling a winner, and I'd run two and four concurrent streams. That's where the cost per token moves, and it's the question a team serving more than one person would ask first.