mini-sglang from Scratch Stage 4 · Faster and bigger

Benchmarks and ablations

With every feature written, the last question is what each optimization is actually worth. This chapter gives two load-testing tools, an offline throughput benchmark and an online streaming client, along with a way to run ablations: turn one optimization off and see what happens to throughput and latency. On a CPU we run the two ablations whose trends are clear; for the ones that need a GPU we give the commands and what to watch.

Self-test: if you can answer these, skip the chapter

1. What question does an offline throughput benchmark answer, and what question does an online load test answer?
2. What affects TTFT and what affects TPOT? Which optimizations mainly improve each?
3. Why does going from batch size 1 to 16 raise throughput far less on a CPU than on a GPU?
4. To quantify what overlap scheduling buys, should you use a large model or a small one?

Answers (try it yourself first, then expand)

1. The offline throughput benchmark hands the engine every request at once and measures how many tokens per second it generates at full load (how fast the engine itself is); the online load test sends requests at a given rate and measures TTFT, TPOT and tail latency under that load (whether it meets the SLO).
2. TTFT: queueing time, the prefill work (prompt length, prefix hit rate), the scheduling policy. TPOT: the time per decode step (batch size, KV length, kernel efficiency, CPU overhead). Prefix caching and chunked prefill mainly improve TTFT; CUDA Graph, overlap scheduling and better attention kernels mainly improve TPOT.
3. A CPU's ratio of compute to bandwidth differs from a GPU's: on a CPU even a batch of 1 already spends a substantial share on compute, rather than almost entirely on reading weights as a GPU does, so growing the batch brings far less "free" throughput.
4. A small model: overlap scheduling hides a fixed CPU cost, so the smaller the model and the shorter each step on the GPU, the larger that cost's share and the clearer the gain; on a large model the GPU time is long and the CPU cost was already drowned out.

Files you will write: benchmark/offline.py, benchmark/client.py.

Files so far: 82 (3 new, 0 modified, 0 brought in early)

minisgl/
├── __init__.py
├── core.py
├── env.py
├── __main__.py
├── shell.py
├── attention/  (6 files, unchanged)
├── benchmark/
│   ├── __init__.py  ← new in this chapter
│   ├── offline.py  ← new in this chapter
│   ├── client.py  ← new in this chapter
├── distributed/  (3 files, unchanged)
├── engine/  (5 files, unchanged)
├── kernel/  (3 files, unchanged)
├── kvcache/  (5 files, unchanged)
├── layers/  (9 files, unchanged)
├── llm/  (2 files, unchanged)
├── message/  (5 files, unchanged)
├── models/  (7 files, unchanged)
├── moe/  (4 files, unchanged)
├── scheduler/  (9 files, unchanged)
├── server/  (4 files, unchanged)
├── tokenizer/  (4 files, unchanged)
├── utils/  (8 files, unchanged)

This step's main: examples/ch21_benchmark.py — it uses only the files above; python tools/steps.py check rebuilds this tree chapter by chapter and runs it.

Offline throughput

minisgl/benchmark/offline.py · run

def run(model: str, num_seqs: int, max_input_len: int, max_output_len: int,
        dtype: torch.dtype = torch.bfloat16, **llm_kwargs) -> dict:
    seed(0)
    llm = LLM(model, dtype=dtype, **llm_kwargs)
    prompts = [[randint(0, 10000) for _ in range(randint(max_input_len // 10, max_input_len))]
               for _ in range(num_seqs)]
    params = [SamplingParams(temperature=0.6, ignore_eos=True,
                             max_tokens=randint(max_output_len // 10, max_output_len))
              for _ in range(num_seqs)]
    llm.generate(["Benchmark: "], SamplingParams(max_tokens=4))  # warm-up
    t = time.perf_counter()
    outputs = llm.generate(prompts, params)
    t = time.perf_counter() - t
    total = sum(len(o["token_ids"]) for o in outputs)
    llm.shutdown()
    return {"output_tokens": total, "seconds": t, "throughput": total / t}

The same approach as the official benchmark/offline/bench.py (which came from nano-vllm): generate a number of random requests with input and output lengths uniform within a range, use ignore_eos=True so every request generates its full max_tokens, and measure the output-token throughput. It measures how fast the engine can go at full load and says nothing about any single request's latency.

python -m minisgl.benchmark.offline --model Qwen/Qwen3-0.6B --num-seqs 256 \
    --max-input-len 1024 --max-output-len 1024 --page-size 256 --cuda-graph-max-bs 256

Online load testing

minisgl/benchmark/client.py · one_request

async def one_request(client: httpx.AsyncClient, url: str, prompt: str, max_tokens: int) -> RequestResult:
    body = {"model": "m", "messages": [{"role": "user", "content": prompt}], "max_tokens": max_tokens,
            "temperature": 0.0, "ignore_eos": True, "stream": True}
    res = RequestResult()
    start = time.perf_counter()
    async with client.stream("POST", url, json=body) as resp:
        async for line in resp.aiter_lines():
            if not line.startswith("data: ") or line == "data: [DONE]":
                continue
            delta = json.loads(line[6:])["choices"][0]["delta"].get("content")
            if delta:
                now = time.perf_counter() - start
                if not res.chunk_times:
                    res.ttft = now
                res.chunk_times.append(now)
                res.output_chars += len(delta)
    res.latency = time.perf_counter() - start
    return res

The online load test imitates real users: requests arrive as a Poisson process (exponentially distributed gaps), responses stream, and it records each request's TTFT (from sending to the first chunk), TPOT (the average gap between the chunks after that) and end-to-end latency, reporting the mean, the median and P99. It answers what the experience is like at a given request rate.

python -m minisgl --model Qwen/Qwen3-0.6B &
python -m minisgl.benchmark.client --num-requests 256 --rate 8 --max-tokens 256

Raise --rate step by step, plot request rate against P99 TTFT / P99 TPOT, and the highest rate that still meets the latency target (the SLO) is this machine's capacity (the method is in the load testing and capacity planning chapter of the Inference Systems handbook).

Two ablations on a CPU

examples/ch21_benchmark.py

"""Chapter 21: two ablations on a CPU, the continuous-batching batch size and the prefix cache.

The absolute numbers on a CPU mean nothing, but the trends match a GPU's. How to measure on a GPU
is in the text.
"""

import time
from random import randint, seed

import torch
from minisgl.core import SamplingParams
from minisgl.llm import LLM

PATH = "models/Qwen3-0.6B"
seed(0)
prompts = [[randint(100, 10000) for _ in range(randint(32, 64))] for _ in range(16)]
params = SamplingParams(max_tokens=32, ignore_eos=True, temperature=0.6)

print("1. Max concurrent requests (16 requests, 32-64 in, 32 tokens out)")
for max_running in (1, 4, 16):
    llm = LLM(PATH, dtype=torch.bfloat16, max_running_req=max_running, num_page_override=4096,
              max_seq_len_override=256)
    llm.generate([[1, 2, 3]], SamplingParams(max_tokens=2))  # warm-up
    t = time.perf_counter()
    out = llm.generate(prompts, params)
    dt = time.perf_counter() - t
    n = sum(len(o["token_ids"]) for o in out)
    print(f"  max_running_req={max_running:<3} took {dt:6.2f}s  output throughput {n / dt:7.1f} token/s")
    llm.shutdown()

print("2. Prefix cache (16 requests sharing a 400-token prefix, 16 tokens each on top)")
shared = [randint(100, 10000) for _ in range(400)]
prompts2 = [shared + [randint(100, 10000) for _ in range(16)] for _ in range(16)]
for cache in ("naive", "radix"):
    llm = LLM(PATH, dtype=torch.bfloat16, max_running_req=16, num_page_override=16384,
              max_seq_len_override=512, cache_type=cache)
    llm.generate([[1, 2, 3]], SamplingParams(max_tokens=2))
    computed = []
    orig = llm._forward
    llm._forward = lambda fi: (computed.append(len(fi.batch.positions)) if fi.batch.is_prefill else None,
                               orig(fi))[1]
    t = time.perf_counter()
    llm.generate(prompts2[:1], SamplingParams(max_tokens=1))  # the first request computes the prefix
    llm.generate(prompts2[1:], SamplingParams(max_tokens=8, ignore_eos=True))
    dt = time.perf_counter() - t
    print(f"  {cache:<5} prefill computed {sum(computed):>5} tokens, total {dt:5.2f}s")
    llm.shutdown()

What it prints

1. Max concurrent requests (16 requests, 32-64 in, 32 tokens out)
  max_running_req=1   took  71.73s  output throughput     7.1 token/s
  max_running_req=4   took  50.31s  output throughput    10.2 token/s
  max_running_req=16  took  34.52s  output throughput    14.8 token/s
2. Prefix cache (16 requests sharing a 400-token prefix, 16 tokens each on top)
  naive prefill computed  6656 tokens, total 117.02s
  radix prefill computed   656 tokens, total 18.69s

Continuous batching: with the same 16 requests, allowing only 1 to run at a time gives about 21 tokens/s, and allowing 16 gives roughly twice that. On a GPU the gap would be far larger: decode is memory-bound, so going from a batch of 1 to 16 leaves the time per step almost unchanged (reading the weights dominates) and throughput grows nearly linearly, whereas a CPU's compute is much weaker relative to its bandwidth, so compute becomes the bottleneck as soon as the batch grows and the gain is squeezed.

Prefix caching: with 16 requests sharing a 400-token prefix, the naive cache computes that prefix 16 times (6656 tokens); the radix cache computes it once, after which each request computes only its own 16 tokens plus the last prefix token (656 in total), and the time drops from about 18 seconds to about 4. What prefix caching buys is proportional to "shared prefix length × request count", which is substantial in multi-turn chat, agents and few-shot prompting.

The ablation list for a GPU

Ablation	Command	What to watch
overlap scheduling	MINISGL_DISABLE_OVERLAP_SCHEDULING=1	offline throughput; the gap is clearest with a small model and a large batch
CUDA Graph	--cuda-graph-max-bs 0	TPOT; the smaller the batch the larger the gap
prefix caching	--cache-type naive	TTFT and throughput on a shared-prefix workload
attention backend	--attn fi / --attn fa / --attn fa,fi	prefill throughput (FA3 is strong on Hopper), decode TPOT (FlashInfer is strong)
chunk size	--max-prefill-length 2048 / 16384	TTFT on long prompts and the TPOT spike during decode
page size	--page-size 1 / 16 / 256	prefix hit rate (coarser with bigger pages) against management overhead

Change one thing at a time in an ablation and leave everything else alone; run each configuration three times and take the median. To see where the time actually goes, capture a timeline with Nsight Systems: the gaps between CPU scheduling and GPU kernels are exactly what overlap scheduling and CUDA Graph exist to remove (the method is in the profiling chapter of the Inference Systems handbook).

The official implementation

• offline: benchmark/offline/bench.py (256 requests, 100-1024 input and output, page_size=256, cuda_graph_max_bs=256)
• online: benchmark/online/bench_qwen.py, which replays the public Qwen request trace from Alibaba Cloud Bailian; the client at benchmark/client.py is about 500 lines and reports more statistics
• the official README gives offline throughput and online latency results on an H200 (the online test compared against SGLang)

Tests

tests/test_ch21_benchmark.py · test_offline_benchmark_runs

def test_offline_benchmark_runs():
    r = run(QWEN3, num_seqs=6, max_input_len=40, max_output_len=16, dtype=torch.float32,
            max_running_req=8, num_page_override=1024, max_seq_len_override=128)
    assert r["output_tokens"] > 6 and r["throughput"] > 0

How to explain it

On benchmarking: an offline throughput test measures how many tokens per second the engine handles at full load, while an online load test measures TTFT, TPOT and tail latency (P99) at a given request rate, and the two measure different things. TTFT depends on queueing and the prefill work (prompt length, prefix hits), so chunked prefill, prefix caching and prefill/decode disaggregation mainly improve it; TPOT depends on the time per decode step, so CUDA Graph, overlap scheduling, quantization and speculative decoding mainly improve it. Ablations change one thing at a time; overlap scheduling and CUDA Graph pay off most with small models and small batches, where the CPU's share is high, so a small model is the right one to quantify them. On a CPU, going from batch 1 to 16 raises throughput only a little because the CPU's compute is already saturated rather than memory-bound as on a GPU.

Exercises

1. Use benchmark/client.py against a --max-running-requests 4 service on a CPU and raise --rate from 0.5 to 4, watching TTFT. Explain where the knee appears.
2. Design a "multi-turn chat" workload: each user sends 5 turns in a row, with each prompt containing all the earlier turns. Compare radix against naive on TTFT under that load.
3. Reproduce the official offline benchmark on a GPU with overlap scheduling and CUDA Graph each turned off, recording throughput. Which matters more for Qwen3-0.6B? What about Qwen3-14B? Why?

Answers

1. Once the arrival rate exceeds the service's capacity (the throughput max_running_req concurrent requests can provide), requests pile up in the waiting queue and TTFT jumps from "prefill time" to "queueing plus prefill time" and keeps growing with time. That is queueing theory's latency explosion as utilization approaches 1.
2. In multi-turn chat each turn's prompt prefix is the previous turn's complete conversation, so the radix cache hits almost the whole prefix and TTFT depends only on the new part, while naive prefills from scratch every turn and TTFT grows linearly with the turn number.
3. A small model's GPU time per step is very short, so the CPU cost and kernel-launch overhead are a large share and both optimizations show clear gains; a large model's step is long, the overheads are a small share, and the gains shrink. Which one matters more depends on the batch size: the smaller the batch, the larger the share of launch overhead that CUDA Graph removes.

Summary

• Offline throughput measures engine speed at full load; online load testing measures TTFT, TPOT and tail latency at a given request rate.
• A CPU is enough to see what continuous batching and prefix caching buy; the former is far clearer on a GPU.
• Ablations change one thing at a time, and overlap scheduling and CUDA Graph pay off most with small models and small batches.

Found a mistake or a better way to say it? Edit this page on GitHub · Report an issue