Tuning GLM-5.1 on eight H200s: three speculative tokens beat two by 29%

In April I rented a node with eight H200s to run GLM-5.1 for my own coding agent. I wasn't chasing a leaderboard. I wanted to know whether a big open model could feel fast enough for one person typing at it. The first benchmark after boot took 45.8 seconds to show a token. Less than two hours later it was under a second, at 22 tokens per second.

The node, the spot pricing and the private endpoint are in the case study. This post covers the tuning runs, including the two passes that errored and the numbers I didn't record.

The benchmark is small on purpose

Every measurement here comes from benchmark JSON written on 2026-04-12, in UTC; the prices and vLLM's published numbers are from the repo's planning notes. Each pass ran two tests against the same endpoint my agent used.

The first is a single-stream benchmark with thinking off: a 21-token prompt, answer streamed back. The answer length varied from 665 to 741 tokens between runs, and the best run streamed 681. The second is a reasoning prompt with thinking on: 138 prompt tokens, and 2,048 completion tokens every time.

Tokens per second here means completion tokens divided by wall time, and wall time includes the wait for the first token. It's what I felt at the keyboard, not the decode rate the server prints in its log.

The first run was slow, and I can't fully explain it

The first benchmark after boot ran at 02:28. It took 45.8 seconds to show a token and averaged 8.44 tokens a second. The reasoning prompt two minutes later waited 147.3 seconds for its first visible token.

I didn't record that server's flags. It was a fresh server with a config I can't reconstruct, so I can't pin the gap on any one setting.

Two flags stayed fixed for every run after that:

• VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0. On the H200s, FlashInfer's FP8 block-scale GEMM path tried to JIT-compile kernels with nvcc and crashed the workers on the first real completion. With it off, the FP8 linear layers run on DeepGEMM and CUTLASS kernels instead.
• VLLM_DEEP_GEMM_WARMUP=skip, which skips DeepGEMM's warmup at startup. I set it to keep restarts between candidates short. I never timed startup with and without it, so I can't tell you what it saved.

Three configs, one change at a time

MTP is GLM-5.1's multi-token prediction head. It drafts the next few tokens cheaply, and the full model checks the whole draft in one forward pass. Each accepted draft token is a token that didn't need its own full decode step. "Eager" means CUDA graphs are off.

Three speculative tokens beat two by 29% on the single stream, 22.25 against 17.29. Time to first token barely moved, which fits: speculation speeds up decoding, and the first token comes from prefill. On the reasoning prompt the gap was smaller, 14.8 against 13.62.

CUDA graphs barely mattered

I expected CUDA graphs to help. They cut kernel launch overhead, and decoding one stream is mostly small kernels. With MTP-3 they measured 21.65 tokens a second, against 22.25 without them.

That gap is smaller than the run-to-run noise. Eager MTP-3 ran six times over about an hour, and its single-stream results ranged from 21.76 to 22.49. So I read the graphs result as no measurable difference on this test, not as graphs being slower. With a single graphs run, that's as far as the data goes.

The loop kept the 22.25 run as its incumbent, and that's the figure I quote elsewhere. It isn't the highest single run: 22.47 came from a pass that errored later, which is the next section, and 22.49 from a benchmark the loop never finished scoring.

Two of seven passes errored

The loop recorded seven passes. Two failed in the Terminal-Bench step, after the throughput tests had finished. One was killed with SIGTERM, and the other exited with status 1 on a one-task run.

The loop gives a failed pass a huge negative score so it can never become the incumbent. That's the right default. It also meant those passes' throughput numbers never reached the summary file, and I only found them later in the per-step JSON. If a loop has several stages, I'd now store each stage's result whatever happens to the stages after it.

The score itself is a weighted sum: 10 × tokens/s, plus 5 × reasoning tokens/s, minus seconds to the first token. A Terminal-Bench term stayed empty in these runs. Eager MTP-3 scored 295.5, graphs with MTP-3 scored 284.6 and eager MTP-2 scored 240.1.

Thinking costs most of the wait

With thinking on, the first visible token took 86.8 seconds on the best config. The model was working the whole time; reasoning tokens just aren't shown. For an interactive agent that's a long time to look at a spinner, and the reasoning run decoded slower too, at 14.8 tokens a second.

My planning estimate was 15–30 tokens a second with thinking on, and I landed just under it, at 14.8. With thinking off I'd estimated 30–45 and missed.

What one stream costs

At 22.25 tokens a second, one stream produces 80,100 tokens an hour. The node cost $9.44 an hour on spot and $18.80 on-demand, which works out to about $118 per million output tokens on spot and $235 on-demand.

vLLM's published benchmark for this model and node shape reaches 526.57 output tokens a second with 32 concurrent requests. I didn't test concurrency myself, so I can't tell you what my setup would do there.

The limits of this data

One stream. Everything here is a single user. Speculative decoding usually gains less as batches grow, because the GPU has less spare compute for checking drafts. I didn't test that.

One prompt per test. A 21-token prompt and one reasoning prompt aren't a workload. Code-heavy output could accept drafts at a different rate than this answer did.

No acceptance rates. vLLM logs speculative decoding metrics, but the loop didn't store them. I can say MTP-3 was faster than MTP-2, not how many of its drafts were accepted, so I can't tell you whether MTP-4 would help or hurt.

What I'd run next

I'd store the acceptance rate on every pass, swap the single prompt for a few dozen taken from real agent sessions, and try MTP-4 once the acceptance data says it's worth trying. Then I'd run two and four streams at once. A single stream answered the question I had, which was whether this felt usable for me. It doesn't answer whether it's a good way to serve a team.