Zac Peterson Contact
Menu

LLM inference

Faster LLM serving on the GPUs you already pay for

Most slow or expensive inference setups I see aren't short on hardware. They're running a default config, a batch size nobody chose, or a node shape that made sense for a different model. I find out which, fix it, and show you the numbers.

Speculative tokens
Real numbers from GLM-5.1 FP8 on eight H200s, one stream: MTP-2 ran at 17.29 tokens/s and MTP-3 at 22.25. The acceptance rate in the animation is illustrative.

What I look at

  • Time to first token and tokens per second, per stream and in aggregate, at your real concurrency.
  • Speculative decoding (MTP or a draft model) and how many tokens to speculate.
  • Precision: FP8 and other quantized formats, with an eval to catch quality loss.
  • Batching and scheduler settings in vLLM or SGLang.
  • Node shape, GPU count, and spot versus on-demand pricing.
  • Compile and warmup caches, so a restart doesn't recompile kernels or download the weights again.

The first step is always a benchmark that looks like your traffic. Prompt length, output length and concurrency change which setting matters, and a benchmark that gets them wrong will happily tell you to tune the wrong thing.

A recent example

In April I ran GLM-5.1 in FP8 on a rented node with eight H200s, for my own coding agent. With two speculative tokens (MTP-2), one stream ran at 17.29 tokens per second. Three (MTP-3) took it to 22.25. Turning CUDA graphs on gave 21.65, slightly slower, so I left them off. The node cost $9.44 an hour on spot and $18.80 on demand.

That's one stream on one prompt, so it says nothing about throughput under load. It does show that one setting, the number of speculative tokens, moved single-stream speed by 29%.

I've also deployed Kimi K2.7 Code with SGLang on a Google Cloud GPU host.

The case study covers the setup: spot capacity, fast boots and a private endpoint. The write-up has every tuning run, including the ones that errored.

What you end up with

A benchmark you can rerun, the config changes with before-and-after numbers, and a short note on what to watch.

Case studies and write-ups

Questions people ask

Self-hosted only?

Mostly self-hosted open models, on your GPUs or a cloud's. If you're on a hosted API, I can help you measure latency and cost and work out whether self-hosting would pay off.

Which serving engines?

vLLM and SGLang. I've run both on rented GPU nodes.

Can it run on Kubernetes?

Yes. Once GPU nodes, images and caches are set up properly, an inference server is just another workload.

Get in touch

If something here sounds like your problem, send me a few lines about what you're running.

zac@zacp.xyz