mini-sglang from Scratch

mini-sglang is the SGLang team's "teaching SGLang": about 5,000 lines of core Python that still contain everything a modern inference engine needs, from radix cache, chunked prefill, overlap scheduling and tensor parallelism to FlashInfer / FlashAttention, CUDA Graph and an OpenAI-compatible online service. This handbook starts from a 126-line minimal engine that runs and checks its own answers, then builds upward one file per chapter following the official module layout, with a running engine in hand at every step, until you have a complete, working mini-sglang whose interfaces match the official one. Every step has tests, and the output matches Hugging Face transformers token by token.

30 chapters19 exercisespytest, token-for-token against HF transformers

Start reading→RoadmapExercises

What you will be able to do

• Explain mini-sglang's process layout: the API server, the tokenizer, the detokenizer, one scheduler process per TP rank, and every kind of message they pass over ZMQ;
• Implement the compute side yourself: an op and weight system that does not rely on nn.Module, streaming weight loading, a paged KV pool, the page table and the token pool, a pluggable attention backend, and the sampler;
• Implement the scheduling side yourself: continuous batching, admission control, page allocation and release, the radix cache (matching, insertion, splitting, LRU eviction, reference counting), chunked prefill and overlap scheduling;
• Explain and verify by hand the three mechanisms that make it fast: the four ways to shard a linear layer under tensor parallelism, FlashInfer's plan/run and FlashAttention's paged interface, and CUDA Graph's fixed-buffer contract;
• Read every file in the official repository, and explain the five problems this reimplementation found and fixed: why each one happens, how to reproduce it, and how to fix it.

Learning path: start from a skeleton that runs

The chapter-by-chapter path across all the handbooks is in the roadmap. This book belongs after "Building an inference engine from scratch" in the Inference Systems handbook: nano_engine there builds the concepts, and here a real, complete code base at production performance turns every detail into code.

This book is not "build the parts first, assemble at the end". Step 0 gets a real engine running in under two hundred lines (read the weights, write the forward pass by hand, a naive KV cache, greedy decoding, matching Hugging Face token by token), and then four stages each replace one naive part of it. At the end of every stage you hold an engine that is stronger than the last, still runs, and still matches Hugging Face.

Step 0 assumes you already know what Qwen3's forward pass computes; if RMSNorm, RoPE, GQA and the KV cache are still new to you, read Transformer by Hand first — it writes Qwen3.5's hybrid architecture block by block from the weight file, and by its chapter 10 you hold a stateful single-process engine that leads straight into this book.

Stage	Chapters	What you hold at the end of the stage	Suggested time
Overview and step 0	Architecture · Step 0: a minimal engine	A 126-line engine that imports nothing from minisgl: it generates, checks its answers and measures tokens/s; plus a table of "what it lacks → which chapter adds it"	1 day
Stage 1 · Computing it right	Core data structures · Op layer · Model and weights · KV pool · Attention backend · Engine and sampling	The skeleton replaced by the real structure: paged KV, pluggable attention backends, a batched sampler; assemble a batch by hand and run prefill + decode with unchanged output	4-5 days
Stage 2 · Scheduling it well	Scheduler skeleton · CacheManager · Radix cache · Chunked prefill · Overlap scheduling	Many requests at once: continuous batching, prefix reuse, long prompts in chunks, CPU cost hidden; the offline LLM.generate interface works	1 week
Stage 3 · Serving	Messages and ZMQ · Tokenizer · Scheduler IO · API server	A multi-process OpenAI-compatible service: usable with curl, with streaming and abort on disconnect	3-4 days
Stage 4 · Faster and bigger	Tensor parallelism · GPU attention · CUDA Graph · CUDA kernels · MoE · Benchmarks	Many GPUs, GPU kernels, MoE models; a benchmark table with a number for every optimisation	1 week
Wrapping up	The gap to SGLang	Know what is still missing and pick a direction to keep going	as needed
Projects	The GPU performance bar · Adding the hybrid model Qwen3.5	Reach 60% of the official build on a GPU; give the engine a linear-attention layer with per-request state	1-2 weeks each

How this book works

One file per chapter, and every step runs. Each chapter opens with the tree of files "so far": the ones this chapter creates, the ones it modifies, and the ones brought in early so that its main can run are all marked; the main at the end of the chapter uses only the files in that tree. python tools/steps.py check rebuilds the package chapter by chapter from that tree and runs each chapter's main — it guarantees that no chapter quietly uses anything from a later one.

Same names, same interfaces. Our package is also called minisgl, and the directory layout, class names, function names and method signatures all match the official one (the official repository is pinned to commit 9a91cfa of 2026-05-17). Each chapter opens with the files you are about to write, and the !!! upstream boxes in the text give the exact location of the matching official code, with line-numbered links, so after a chapter you can diff your version against the official one line by line.

Every step is verifiable on a CPU. The official implementation only supports NVIDIA GPUs. To let any machine follow along and to make every step checkable, we did three things:

1. Added a very thin device abstraction (minisgl/utils/device.py): real streams, events and pinned memory on CUDA, no-ops on CPU;
2. Gave every GPU op a PyTorch reference implementation: the attention backend has an extra torch option, and sampling, RMSNorm, RoPE and the KV-cache writes all have CPU versions;
3. Verified the GPU-only parts on a CPU through stand-ins with the same interface: FlashInfer and FlashAttention each have a PyTorch fake with the same interface, CUDA Graph has a CPU emulation (which forces every input through fixed buffers, so missing one copy makes the result wrong), CUDA kernels run on the CPU simulator from the CUDA handbook, and Triton kernels run in interpreter mode.

On a real GPU the same code switches to FlashInfer, FlashAttention, CUDA Graph and the custom kernels by itself.

Every step matches Hugging Face. Besides the unit tests, each chapter has an end-to-end check: run Qwen3-0.6B in float32 and have every greedy-decoded token match transformers's generate. By the end the same check covers the radix cache, chunked prefill, overlap scheduling, page sizes above 1, tensor parallelism (TP=2 and TP=4), all three attention backends and CUDA Graph, across the Qwen2.5, Llama 3 (long-context RoPE) and Qwen3-MoE architectures.

Where we differ from the official implementation

Besides the CPU support above there are a few differences, each explained in a "difference from upstream" box in its chapter. Five of them fix problems in the official code, and each has a test that fails if you put the official version back: the finished flag arriving one token early under overlap scheduling, one stale message sent after EOS, a request slot reused while its batch is still on the GPU, and KV pages freed twice when a request aborts mid-prefill (chapter 11), plus a PUB/SUB broadcast that does not wait for its subscribers (chapter 12). The rest are simplifications: the model definitions are merged into one generic decoder file; tensor parallelism uses only torch.distributed, without the official PyNCCL built on tvm-ffi; and the fused-MoE Triton kernel is a simplified version.

The code, and running it

All the code is in the repository's minisgl/ directory:

minisgl/
├── python/minisgl/      our implementation, one to one with the official python/minisgl/
├── tests/               one test file per chapter (test_ch07_scheduler.py …), plus GPU-library fakes and a CUDA self-check
├── examples/            the script behind every "what it prints" block in the text
├── tools/check.py       one command runs every check: examples, nvcc build, CPU simulator, pytest
├── tools/diagrams.py    generates the book's 15 architecture figures (SVG, following the light / dark theme)
├── tools/videos/        scene scripts and renderer for the 6 animations (edge-tts voice, frames then ffmpeg)
└── docs/                this handbook

cd minisgl
pip install -e ".[dev]"                                   # deps: torch, transformers, pyzmq, fastapi and friends
python -m minisgl --model Qwen/Qwen3-0.6B                 # start an OpenAI-compatible service (GPU or CPU)
python -m minisgl --model Qwen/Qwen3-0.6B --shell        # interactive chat
PYTHONPATH=python:tests pytest -q tests                   # the whole test suite (needs Qwen3-0.6B downloaded into models/)

The "files you will write" at the start of each chapter is what you type yourself. A good rhythm: read the idea, write your own version, then compare against this book's implementation and the official one, and finally run the chapter's tests. If you cannot answer a chapter's opening self-test, go back to the matching concept chapter in the Inference Systems handbook.

The book has 15 architecture figures and 6 narrated, subtitled animations of 1.5 to 2 minutes each, worth watching before you read a chapter: the life of a request, continuous batching and admission control, the radix cache, overlap scheduling, tensor parallelism and CUDA Graph. The narration and subtitles are in Chinese.

Found a mistake or a better way to say it? Edit this page on GitHub · Report an issue