mini-sglang from Scratch Overview and step 0

Architecture and the path we follow

Before writing anything, look at the whole system once: a request arrives over HTTP, passes through a set of processes, turns into a set of data structures, gets computed somewhere, and finally flows back to the client one token at a time. This chapter gives the whole picture of mini-sglang, the dependencies between its modules, the order in which we rebuild it, and how to verify every step on a machine with no GPU.

Self-test: if you can answer these, skip the chapter

1. What kinds of processes make up mini-sglang? How many processes does a TP=4 service have in total?
2. Inside a scheduler process, what does the "scheduler" do and what does the "engine" do?
3. Why does the model's forward() take no arguments?
4. The official implementation only supports CUDA. Which three means does this book use to verify GPU-related code on a CPU?

Answers (try it yourself first, then expand)

1. Three kinds: the API server (the HTTP entry point), the tokenizer / detokenizer process, and one scheduler process per TP rank (each holding a scheduler and an engine). At TP=4 that is 1 + 1 + 4 = 6 processes.
2. The scheduler decides what to compute this step and where the KV goes: it receives requests, picks the batch, allocates KV and handles the results. The engine does the computing: it runs the model's forward pass and the sampling.
3. Everything about the current batch (requests, positions, attention metadata) lives in a global Context, and each model layer reads it from there when it needs it, so forward() needs no arguments.
4. The device abstraction (the same code runs on a CPU), reference implementations and stand-ins with the same interface as the GPU ones (the attention backend, the fake FlashInfer / FlashAttention), and simulators (the CUDA Graph emulation, the CPU simulator for CUDA kernels, the Triton interpreter).

The result first

This is what we are going to build: the offline LLM interface running Qwen3-0.6B on a CPU, generating for three requests at once.

examples/ch00_quickstart.py

"""The overview: run a few requests through the offline interface to see what we are building."""

import time

import torch
from minisgl.core import SamplingParams
from minisgl.llm import LLM

llm = LLM("models/Qwen3-0.6B", dtype=torch.float32, max_running_req=8,
          num_page_override=2048, max_seq_len_override=512)
prompts = ["The capital of France is", "List three prime numbers:", "def fibonacci(n):"]
t = time.perf_counter()
outputs = llm.generate(prompts, SamplingParams(max_tokens=16, ignore_eos=True))
elapsed = time.perf_counter() - t
for p, o in zip(prompts, outputs):
    print(f"{p!r} -> {o['text']!r}")
print(f"3 requests x 16 tokens, {elapsed:.2f} s on a CPU")
llm.shutdown()

What it prints

'The capital of France is' -> ' Paris. The capital of Italy is Rome. The capital of Spain is Madrid.'
'List three prime numbers:' -> ' 2, 3, 5, 7, 11,'
'def fibonacci(n):' -> '\n    if n == 0:\n        return 0\n    elif n'
3 requests x 16 tokens, 2.00 s on a CPU

The online service is a single command, python -m minisgl --model Qwen/Qwen3-0.6B, after which any OpenAI client can talk to http://127.0.0.1:1919/v1/chat/completions.

The processes

mini-sglang is a multi-process system. At TP=2 it looks like this:

mini-sglang's processes (TP=2)

• API server (the main process): takes HTTP requests, gives each one a uid, and sends the text to the tokenizer; it then streams the delta text it gets back to the client over SSE.
• tokenizer / detokenizer: tokenization and incremental detokenization. By default both share one process (--num-tokenizer 0).
• scheduler processes: one per TP rank. Each holds a Scheduler (which decides which requests run this step and allocates KV cache) and an Engine (which owns the model and the KV pool and runs the forward pass and sampling). Only rank 0 talks to the tokenizer; it broadcasts every message it receives verbatim to the other ranks, so all ranks make exactly the same scheduling decisions.

So a TP=4 service has 1 + 1 + 4 = 6 processes in total (API server, tokenizer, four schedulers). Control messages between processes go over ZMQ; tensor-parallel data goes over NCCL.

The official implementation

The startup logic is in server/launch.py · launch_server (lines 40-113) and the scheduler's main loop in scheduler/scheduler.py · Scheduler.overlap_loop (lines 83-106). The official docs/structures.md has a process diagram that matches the one above.

The journey of one request

the processes and messages a request passes through

1. The client sends POST /v1/chat/completions. The API server assigns uid = 7 and sends TokenizeMsg(uid=7, text=[messages], sampling_params).
2. The tokenizer applies the chat template, tokenizes, and sends UserMsg(uid=7, input_ids=tensor) to scheduler rank 0.
3. The scheduler puts it in the prefill waiting queue. On some scheduling step PrefillAdder looks up the longest cached prefix in the radix cache, checks whether there is enough memory left and a free request slot, and admits it if so: it allocates a page-table row and locks the prefix it hit.
4. _prepare_batch allocates KV pages for the tokens to compute this step, works out each token's position and the KV location to write, and the attention backend prepares its metadata.
5. Engine.forward_batch: run the model forward, take the logits at each request's last position, and sample. The sample is written straight back into the token_pool on the GPU, as the next step's input, and is also copied back to the CPU asynchronously.
6. The scheduler handles the result: it appends the new token to the request, decides whether it has finished, and sends DetokenizeMsg(uid=7, next_token, finished). For a request that has just finished prefill, the prompt's KV is inserted into the radix cache.
7. Every decode step after that repeats steps 4 to 6 until EOS or max_tokens. At the end the request slot is freed and the KV handed back to the cache.
8. The detokenizer turns tokens into text incrementally and sends UserReply(uid=7, incremental_output, finished); the API server writes it into the SSE stream.

Every step along this path is a chapter of this book.

Animation: the life of a request (about 2 minutes, Chinese narration and subtitles)

Modules and dependencies

The modules under python/minisgl/, from the bottom of the dependency stack up:

module layering

Module	What it holds	Chapter
utils, env	logging, the registry, ZMQ queues, HF helpers; (ours) the device abstraction	introduced where first used
core	SamplingParams, Req, Batch, Context	Core data structures
distributed	TP info, all-reduce / all-gather	Tensor parallelism
kernel	ops: RMSNorm, RoPE, KV writes, lookups (CPU reference plus GPU versions)	Op layer, CUDA kernels
layers	the BaseOP system and the layers	Op layer
models	model config, model structure, weight loading	Model and weight loading
kvcache	the KV pool and the prefix cache (naive and radix)	KV pool, Radix cache
attention	attention backends: torch (ours), FlashInfer, FlashAttention	Attention backend, GPU attention
moe	MoE backends: the torch reference, the Triton fused MoE	MoE
engine	Engine, Sampler, GraphRunner	Engine and sampler, CUDA Graph
scheduler	the scheduler and its managers	all of part three
message	inter-process messages and serialization	Messages and ZMQ
tokenizer	the tokenizer / detokenizer process	Tokenizer
server, llm	the API server and launcher; the offline LLM interface	API server, Scheduler skeleton

Two designs run through everything and are worth remembering now:

• The global Context. The model's forward() has no arguments: the input tokens, the positions and the attention metadata all hang off "the current batch", and the current batch lives in a process-wide global Context. Any layer that needs them calls get_global_ctx().batch. This keeps the model code very clean, at the price that the model only runs inside the ctx.forward_batch(batch) context.
• The Registry. --attn fa,fi, --cache-type radix and --moe-backend fused on the command line are names, and the registry maps them to implementations. Adding a backend means writing one class and registering one name.

The rebuild plan

This book is not "build the parts first, assemble at the end". It starts at step 0: a 126-line minimal engine that imports nothing from minisgl — read the weights, write the forward pass by hand, a naive KV cache, greedy decoding, matching Hugging Face token by token. Four stages follow, each chapter replacing one naive part of it:

1. Stage 1 · Computing it right (chapters 1–6): the skeleton becomes the real structure — Req / Batch / Context, the op layer, the model and weight loading, the paged KV pool, attention backends, the Engine and the sampler. It ends at the stage 1 milestone: the same prompt, the same 16 tokens, plus what the skeleton could not do — four requests in one batch.
2. Stage 2 · Scheduling it well (chapters 7–11): the scheduler takes over the lines filled by hand in stage 1 — continuous batching, admission control, the radix cache, chunked prefill, overlap scheduling — leaving only LLM.generate. Milestone: 8 requests in one call, a shared prefix computed once instead of four times.
3. Stage 3 · Serving (chapters 12–15): messages, the tokenizer process, the scheduler's IO, the API server — a multi-process OpenAI-compatible service. Milestone: start it, stream from it, hit it with 8 concurrent requests.
4. Stage 4 · Faster and bigger (chapters 16–21): tensor parallelism, GPU attention backends, CUDA Graph, custom kernels, MoE, and benchmarks. Milestone: TP=2 and graph replay leave the output unchanged; speed needs a real card.

One file per chapter. Each chapter opens with the tree of files "so far" (the ones it creates, modifies, or brings in early so that its main can run are all marked), and the main at its end uses only the files in that tree. python tools/steps.py check rebuilds the package chapter by chapter from that tree and runs each chapter's main — it guarantees that no chapter quietly uses anything from a later one.

Every chapter has the same shape: self-test → file tree and this step's main → the principle → our implementation → the official implementation for comparison → tests and output → exercises → summary. At the end of every stage you hold an engine that is stronger than the last, still runs, and still matches Hugging Face.

Verifying GPU code on a CPU

Having no GPU on your development machine is the normal case. There are four kinds of CUDA-only things in the official code, and the book handles each:

What upstream uses	What this book does	Chapter
torch.cuda.Stream, Event, pinned memory	utils/device.py: used as is on CUDA, replaced by do-nothing NullStream and NullEvent on CPU	Engine, Overlap scheduling
FlashInfer / FlashAttention attention kernels	an extra PyTorch reference backend torch; in tests, a fake FlashInfer / fake sgl_kernel with the same interface checks the argument semantics of the official backend code	Attention backend, GPU attention
CUDA Graph	EmulatedGraph on the CPU: replay can only read the fixed buffers, so missing any input copy makes the result wrong	CUDA Graph
custom CUDA / Triton kernels	CUDA kernels are compiled with nvcc 12.9 and 13.4 and self-checked on the CPU simulator from the CUDA handbook; Triton kernels run in interpreter mode	CUDA kernels, MoE

On a GPU the same code picks FlashInfer / FlashAttention by itself (--attn auto), turns on CUDA Graph, and uses the custom kernels and the Triton fused MoE.

Difference from upstream: the device abstraction

The first thing the official Engine.__init__ does is torch.cuda.set_device, and the scheduler creates a torch.cuda.Stream() directly. We replace those with create_stream(device), create_event(device), pin(device) and friends, which behave identically on CUDA. EngineConfig also gains two fields, device and cpu_kv_cache_bytes.

Environment

git clone https://github.com/AnranS/ai-infra-handbooks && cd ai-infra-handbooks/minisgl
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"            # CPU torch is enough; on a GPU also install ".[gpu]"
# model: put it under models/ (we run float32, which matches HF most reliably)
modelscope download --model Qwen/Qwen3-0.6B --local_dir models/Qwen3-0.6B

It is worth cloning the official repository next to it and reading along:

git clone https://github.com/sgl-project/mini-sglang && git -C mini-sglang checkout 9a91cfa

Running it on a Mac

None of the code in this book requires a GPU: without CUDA the Engine picks the CPU, and torch.cuda.Stream / Event / pinned memory all degrade to no-ops (see utils/device.py); FlashInfer and FlashAttention are replaced by same-interface PyTorch fakes, and CUDA Graph by EmulatedGraph. On macOS, install the dependencies and PYTHONPATH=python:tests pytest -q tests just runs — the tensor-parallel chapters use gloo and are still genuinely multi-process. Two caveats:

• MPS is not used. _pick_device only recognises CUDA; everything else is the CPU. That is deliberate — every test here compares against Hugging Face token by token in float32, and MPS is neither precise nor complete enough for that.
• tools/check.py is not for readers. It is the maintainer's full check and needs the upstream repository at a pinned commit plus two nvcc toolchains. pytest is all you need.

Thread count on a CPU

PyTorch on a CPU defaults to one thread per logical core. The matmuls during decode are tiny, with a batch of only a few, and synchronizing 32 threads makes them tens of times slower: on this book's development machine one decode step takes 2.4 seconds with the default and 40 milliseconds once the thread count drops to the physical core count. So our Engine sets the thread count to physical cores / TP size on a CPU, unless you set OMP_NUM_THREADS yourself.

How to explain it

To explain how to "describe the architecture of an inference engine", you can use mini-sglang as the skeleton: an API server (HTTP, the OpenAI interface) plus a tokenizer / detokenizer process plus one scheduler process per TP rank, with control messages over ZMQ and tensors over NCCL, so 1 + 1 + 4 processes at TP=4. Inside a scheduler process the scheduler decides which requests run this step and where the KV goes, and the engine does the computing (model, attention backend, sampling, CUDA Graph); the model reads the current batch from a global Context, which is why forward() takes no arguments. Then walk through one request (tokenize, queue, prefill, decode step by step, detokenize and stream back, free the KV), and give the reason for multiple processes: it gets around the GIL and lets CPU work run alongside GPU compute.

Exercises

1. Count the processes. You deploy a TP=8 service with mini-sglang. How many processes are there? Which pairs exchange control messages and which exchange tensors? Over what?

Answer

One API server + one tokenizer / detokenizer process + eight scheduler processes (one per TP rank, each holding a scheduler and an engine), so ten. Control messages such as requests, tokenized requests and generated results go over ZMQ: API server to tokenizer to scheduler rank 0, with rank 0 forwarding the raw message to the other ranks, and results travelling back from rank 0 through the detokenizer to the API server. Tensors between the eight scheduler processes (the tensor-parallel all-reduces and so on) go over NCCL.

2. Why this order. The path is "compute it right, schedule it well, serve it, make it fast and big". Why not write CUDA Graph and the custom kernels first?

Answer

A performance optimization does not change the result, only how it is computed, so its correctness can only be judged against a version that is already correct: first get the reference implementation matching HF token by token, then check every added optimization (overlap scheduling, CUDA Graph, FlashInfer, custom kernels) against the same greedy output. Do performance first and a wrong answer could be the model, the scheduling or the optimization itself. Scheduling and serving also set the boundaries for the optimizations: CUDA Graph needs its inputs in fixed buffers and overlap scheduling needs the input token to stay on the GPU, and both are much easier once the data structures are settled.

Summary

• mini-sglang = API server + tokenizer/detokenizer + one scheduler process per TP rank; control messages over ZMQ, tensors over NCCL.
• The scheduler decides what runs this step and where the KV goes, the engine does the computing, and the model reads the current batch from the global Context.
• The order: compute it right, schedule it well, serve it, make it fast and big, with every step matching HF token by token.
• GPU-only parts are verified on a CPU through the device abstraction, reference implementations, same-interface stand-ins and simulators.

Found a mistake or a better way to say it? Edit this page on GitHub · Report an issue