SGLang Design Evolution From research code to a usable service (H1 2024)

v0.2: a scheduling core under four thousand lines reaching 3.1 times

On 25 July 2024 SGLang cut v0.2.0 and published a blog post: up to 3.1 times vLLM's throughput on Llama-70B, often matching or beating TensorRT-LLM, with "the core scheduler in under 4000 lines of Python". This chapter reads v0.2.0's code to find where the 3.1 times comes from: CUDA graphs on by default, a dynamically estimated new_token_ratio with a retraction mechanism, the balance between decoding in a run and streaming, and the bench_serving, DeepSeek-V2 and chunked prefill that landed in the days around the release. It is also the watershed where SGLang went from "a paper's runtime" to "an inference engine": the word "language" barely appears in the blog post.

Self-test: if you can answer these, skip the chapter

1. What is the v0.2 blog post's comparison setup? Which files does "under 4K lines" refer to?
2. How are CUDA graphs used in v0.2.0: which batch sizes are captured, what happens to a batch not in the list, and why is FlashInfer required?
3. What is new_token_ratio? How does it change over a service's lifetime? When does a retraction happen?
4. Is chunked prefill a v0.2.0 feature? How does it relate to v0.2?

Answers for the self-test (answer first, then open this)

1. Llama-8B / 70B / 405B on A100s and H100s, in BF16 and FP8, against vLLM 0.5.2 and TensorRT-LLM v0.10.0 as baselines, load-tested with bench_serving.py. "Under 4K lines" means the scheduling core under managers/controller/: tp_worker.py, infer_batch.py, radix_cache.py, schedule_heuristic.py, cuda_graph_runner.py, model_runner.py and the two controllers, about 3000 lines together.
2. At startup it captures one graph (decode only) for each of the batch sizes [1, 2, 4, 8, 16, …, 128]; at run time it rounds the actual batch up to the nearest captured size and fills the extra slots with dummy requests of length 1, truncating the surplus output after the replay. It requires FlashInfer because FlashInfer's decode wrapper supports use_cuda_graph=True (fixed indptr and indices buffers), which the Triton backend had no interface for at the time, so --disable-flashinfer disables CUDA graphs too.
3. It is the coefficient estimating "how many more tokens a running request will generate": on admission it multiplies the remaining max_new_tokens to reserve space. It starts at 0.7, decays by 0.001 per successful decode step down to a floor (0.2 by default, scaled by the context length), and as soon as decoding runs short of memory, some requests are retracted and the coefficient is raised. A retraction happens when check_decode_mem fails at the head of forward_decode_batch, retracting the newest requests by output length back into the waiting queue.
4. No: #800 merged on 29 July and v0.2.0 was cut on the 25th. It is part of the v0.2 series (v0.2.x) and the blog post does not mention it.

A six-panel strip first, then the chapter:

One release's weight

v02-tag.sh

echo "v0.2.0：$(git log -1 --date=short --format='%ad %h %s' v0.2.0 | cut -c1-70)"
echo "v0.1.5 → v0.2.0 的提交数：$(git rev-list --count v0.1.5..v0.2.0)"
echo "这段时间提交最多的作者："; git shortlog -sn v0.1.5..v0.2.0 | head -6

output

v0.2.0：2024-07-25 1a491d00cb Bump version to 0.2.0 (#730)
v0.1.5 → v0.2.0 的提交数：408
这段时间提交最多的作者：
   131  Lianmin Zheng
    66  Liangsheng Yin
    59  Ying Sheng
    22  zhyncs
    17  Cody Yu
    11  Yineng Zhang

Half a year, several hundred commits and a dozen or so main authors — against January's "a paper's code by four people", this is already a community project. The blog's own words: "At LMSYS.org we have been running Chatbot Arena for more than a year… SGLang is already used by Chatbot Arena to serve some models, and by Databricks, several startups and research institutions, having generated trillions of tokens"; "the core scheduler is implemented in under 4K lines of code"; "SGLang's efficient Python batch scheduler scales well and often matches or beats closed-source systems implemented in C++". Count those under-4K lines:

v020-core-lines.sh

total=0
for f in $(git ls-tree -r --name-only v0.2.0 -- python/sglang/srt/managers/controller); do
  n=$(git show "v0.2.0:$f" | wc -l); total=$((total + n)); printf '%5d  %s\n' "$n" "${f#python/sglang/srt/managers/}"
done
echo "合计 $total 行"

output

  240  controller/cuda_graph_runner.py
  935  controller/infer_batch.py
  202  controller/manager_multi.py
  157  controller/manager_single.py
  383  controller/model_runner.py
  272  controller/radix_cache.py
   70  controller/schedule_heuristic.py
  800  controller/tp_worker.py
合计 3059 行

Source one: CUDA graphs on by default

612 "Enable cuda graph by default" of 13 July added cuda_graph_runner.py (173 lines). ModelRunner.init_cuda_graphs decides which batches to capture:

    def init_cuda_graphs(self):
        from sglang.srt.managers.controller.cuda_graph_runner import CudaGraphRunner

        if self.server_args.disable_cuda_graph or self.server_args.disable_flashinfer:
            self.cuda_graph_runner = None
            return

        logger.info(
            f"[gpu_id={self.gpu_id}] Capture cuda graph begin. This can take up to several minutes."
        )
        batch_size_list = [1, 2, 4] + [i * 8 for i in range(1, 17)]
        self.cuda_graph_runner = CudaGraphRunner(
            self,
            max_batch_size_to_capture=max(batch_size_list),
            use_torch_compile=self.server_args.enable_torch_compile,
        )
        try:
            self.cuda_graph_runner.capture(batch_size_list)

[1, 2, 4] + [8, 16, …, 128], and nothing is captured with --disable-flashinfer. Capturing and replaying:

python/sglang/srt/managers/controller/cuda_graph_runner.py @ v0.2.0 L159-184,186-218

        def run_once():
            input_metadata = InputMetadata.create(
                self.model_runner,
                forward_mode=ForwardMode.DECODE,
                req_pool_indices=req_pool_indices,
                seq_lens=seq_lens,
                prefix_lens=None,
                position_ids_offsets=position_ids_offsets,
                out_cache_loc=out_cache_loc,
                return_logprob=False,
                top_logprobs_nums=0,
                skip_flashinfer_init=True,
            )
            input_metadata.flashinfer_decode_wrapper = flashinfer_decode_wrapper

            return forward(input_ids, input_metadata.positions, input_metadata)

        for _ in range(2):
            run_once()

        torch.cuda.synchronize()
        with torch.cuda.graph(graph, pool=self.graph_memory_pool, stream=stream):
            out = run_once()
        torch.cuda.synchronize()
        self.graph_memory_pool = graph.pool()
        return graph, None, out, flashinfer_decode_wrapper
...
    def replay(self, batch: Batch):
        assert batch.out_cache_loc is not None
        assert not batch.return_logprob
        raw_bs = len(batch.reqs)

        # Pad
        index = bisect.bisect_left(self.batch_size_list, raw_bs)
        bs = self.batch_size_list[index]
        if bs != raw_bs:
            self.seq_lens.fill_(1)
            self.position_ids_offsets.zero_()
            self.out_cache_loc.zero_()

        # Common inputs
        self.input_ids[:raw_bs] = batch.input_ids
        self.req_pool_indices[:raw_bs] = batch.req_pool_indices
        self.seq_lens[:raw_bs] = batch.seq_lens
        self.position_ids_offsets[:raw_bs] = batch.position_ids_offsets
        self.out_cache_loc[:raw_bs] = batch.out_cache_loc

        # FlashInfer inputs
        init_flashinfer_args(
            ForwardMode.DECODE,
            self.model_runner,
            self.req_pool_indices[:bs],
            self.seq_lens[:bs],
            None,
            self.flashinfer_handlers[bs],
        )

        # Replay
        self.graphs[bs].replay()
        output = self.output_buffers[bs]

Three things matter: the input buffers (input_ids, req_pool_indices, seq_lens, out_cache_loc, and FlashInfer's kv_indptr and kv_indices) are allocated once at initialisation for the largest batch, and the kernels in the graph always read those addresses; replay copies the actual batch into the front of the buffers and pads the rest (seq_lens with 1 and out_cache_loc with 0, so a dummy request "writes to slot 0" without error) and truncates the output to the actual batch afterwards; and all of the graphs share one memory pool (graph_memory_pool), so graphs for different batch sizes do not take memory twice. Before capturing it runs two warm-up passes (for _ in range(2): run_once()) to let FlashInfer's plan and PyTorch's caching allocator settle.

Figure: padding the batch when replaying a CUDA graph

One decode step is only tens of milliseconds, sometimes only a few, yet it launches hundreds or thousands of kernels; the cost of launching them one by one from Python can be over half of that at a small batch, and a CUDA graph compresses it into one replay(). This is one of the main reasons v0.2 beat vLLM by the widest margin at small batches and low latency (vLLM had CUDA graphs too at the time, but off by default and fitting its block table in a more complicated way).

Source two: a dynamic new_token_ratio and retraction

The first version's admission estimated how much more a running request would generate with a fixed coefficient of 0.4 (chapter two). v0.2.0 turned it into a quantity that adjusts as it runs:

        # Runtime constants: New generation token ratio estimation
        self.init_new_token_ratio = 0.7
        self.base_min_new_token_ratio = 0.2
        self.new_token_ratio_decay = 0.001
        self.new_token_ratio_recovery = 0.05

        # Runtime constants: The threshold (number of tokens) to trigger layer-wise cuda sync.
        # This can improve the speed for large batch sizes during prefill.
        self.layer_sync_threshold = 8192

        # Runtime constants: others
        self.num_continue_decode_steps = 10
        self.retract_decode_steps = 20
        self.flashinfer_workspace_size = 192 * 1024 * 1024

python/sglang/srt/managers/controller/tp_worker.py @ v0.2.0 L337-347,535-553

        available_size = (
            self.token_to_kv_pool.available_size() + self.tree_cache.evictable_size()
        )
        if self.running_batch:
            available_size -= sum(
                [
                    (r.sampling_params.max_new_tokens - len(r.output_ids))
                    * self.new_token_ratio
                    for r in self.running_batch.reqs
                ]
            )
...
    def forward_decode_batch(self, batch: Batch):
        # Check if decode out of memory
        if not batch.check_decode_mem():
            old_ratio = self.new_token_ratio

            retracted_reqs, new_token_ratio = batch.retract_decode()
            self.new_token_ratio = new_token_ratio

            logger.info(
                "decode out of memory happened, "
                f"#retracted_reqs: {len(retracted_reqs)}, "
                f"#new_token_ratio: {old_ratio:.4f} -> {self.new_token_ratio:.4f}"
            )
            self.forward_queue.extend(retracted_reqs)
        else:
            self.new_token_ratio = max(
                self.new_token_ratio - self.new_token_ratio_decay,
                self.min_new_token_ratio,
            )

The estimation formula on admission is unchanged (available = free + evictable − Σ remaining x the coefficient); what changed is the coefficient: it starts conservative (0.7) and decays by 0.001 with every successful decode step down to the floor min_new_token_ratio (scaled by the model's context length, 0.2 by default) — the longer the requests run, the more convinced the system is that most of them will not use up their max_new_tokens, and the more it dares to admit. There is a fallback when the estimate fails: when check_decode_mem finds that decoding cannot allocate slots, retract_decode retracts some requests (the newest, ordered by output length) back into the waiting queue and raises the coefficient (new_token_ratio_recovery). The retraction mechanism existed in #36 "Fix the possible bug of decode out of memory" of 20 January; v0.2.0 closed the loop between it and the coefficient's rise and fall.

This is where SGLang's and vLLM's scheduling philosophies differ most concretely: vLLM is "admit while it allocates, preempt when it runs short" and SGLang is "estimate the future, admit conservatively, retract when the estimate is wrong". Both avoid deadlock; SGLang's preempts less when request lengths are similar and leans more on the coefficient's adaptation when they vary widely.

Source three: the balance between decoding in a run and streaming

    def forward_step(self):
        new_batch = self.get_new_prefill_batch()

        if new_batch is not None:
            # Run a new prefill batch
            self.forward_prefill_batch(new_batch)
            self.cache_filled_batch(new_batch)

            if not new_batch.is_empty():
                if self.running_batch is None:
                    self.running_batch = new_batch
                else:
                    self.running_batch.merge(new_batch)
        else:
            # Run a decode batch
            if self.running_batch is not None:
                # Run a few decode batches continuously for reducing overhead
                for _ in range(global_config.num_continue_decode_steps):
                    self.num_generated_tokens += len(self.running_batch.reqs)
                    self.forward_decode_batch(self.running_batch)

                    # Print stats
                    if self.tp_rank == 0 and self.decode_forward_ct % 40 == 0:
                        self.print_stats()

                    if self.running_batch.is_empty():
                        self.running_batch = None
                        break

                    if self.out_pyobjs and self.running_batch.has_stream():
                        break
            else:
                self.check_memory()
                self.new_token_ratio = global_config.init_new_token_ratio

The first version's hard-coded "10 decode steps in a row" became global_config.num_continue_decode_steps (still 10), but with an exit added: if self.out_pyobjs and self.running_batch.has_stream(): break — with a streaming request and some output accumulated, it leaves early to send. That is the compromise between throughput (fewer returns to Python, fewer request pickups) and streaming latency; with no new requests it also runs check_memory as a self-check and resets the coefficient to its initial value. What really killed "N steps in a row" is October's overlapped scheduling (chapter 12): once the CPU's work hides behind the GPU, returning to Python every step no longer matters.

The two weeks around the release

v0.2.0 is not an isolated tag, and the commits in the fortnight on either side show what the authors were busy with:

v020-commits.sh

for h in 665815969a c126a6ccba d774acad5c eedc12e12e 679ebcbbdc cdcbde5fc3 7cd4f244a4 e1eae1fd15; do
  git log -1 --date=short --format='%ad  %h  %s' $h | cut -c1-96
done

output

2024-07-13  665815969a  Enable cuda graph by default (#612)
2024-07-20  c126a6ccba  feat: add benchmark serving (#657)
2024-07-18  d774acad5c  Remove the dependency of rpyc (#646)
2024-07-21  eedc12e12e  Support Deepseek MoE Model (#689)
2024-07-26  679ebcbbdc  Deepseek v2 support (#693)
2024-07-29  cdcbde5fc3  Code structure refactor (#807)
2024-07-29  7cd4f244a4  Chunked prefill (#800)
2024-08-05  e1eae1fd15  Support MLA for DeepSeek-V2 with Triton - step 1 (#905)

• #657 bench_serving (20 July): the blog's numbers come from this script (an adaptation of vLLM's benchmark_serving.py, whose random dataset vLLM later took from it in turn). Only with one load-testing tool is "3.1 times" comparable.
• #689 DeepSeek MoE and #693 DeepSeek-V2 (21 and 26 July): the next phase's protagonist arrives early, and the Triton implementation of MLA started on 5 August (chapter 11).
• #800 chunked prefill (29 July): a long prompt no longer monopolises a step but is cut into chunks of --chunked-prefill-size and mixed with decoding; the admission logic in schedule_heuristic.py changed with it. This is one of the items in the Q3 2024 roadmap (issue #634).
• #807 the directory restructuring (the same day): paving the way for scheduler.py to stand alone.

The blog's closing preview is "long context and MoE optimisation", which is exactly v0.3's subject.

Design trade-offs

• CUDA graphs for decode only, and tied to FlashInfer. Prefill's shapes vary too much to be worth capturing; and for a graph to be replayable, the attention backend has to accept fixed buffers — a requirement that later became every attention backend's interface contract (init_cuda_graph_state, init_forward_metadata_capture_cuda_graph).
• Estimate and retract rather than preempt. Simple to implement (a retraction is a return to the queue, with the KV still in the tree to hit again), but the coefficient is global and too coarse for a mixed workload; --schedule-conservativeness was added later for users to tune it.
• A Python scheduler. The blog makes "even a Python implementation can match C++" a selling point. The price only shows at a very large batch where a step is a few milliseconds, which is the problem v0.4's zero-overhead scheduling set out to solve.

What happened afterwards

• CUDA graphs: the list of captured batches, the padding and the shared memory pool are still in use; 2025 added piecewise CUDA graphs (#11490, which also graphs prefill's non-attention parts) and graphs for speculative decoding.
• The new_token_ratio family of parameters is still in global_config.py and schedule_policy.py, and retract_decode is still the first response to running short of memory.
• num_continue_decode_steps lost its meaning once overlapped scheduling became the default and the logic was removed.
• bench_serving.py became one of the load-testing tools SGLang and vLLM share, and it is the script the inference-systems handbook's load-testing chapter covers.

Exercises

1. What padding costs. From replay's implementation, which graph does a batch of 65 replay? How much computation is wasted? What kind of batch list would waste less without letting the capture time and the memory get out of hand?

Answer

It rounds up to 72 (the list is 1, 2, 4, 8, 16, …, 128, in steps of 8), so 7 dummy requests waste about 10% of the decode's computation. A denser list (small steps at small batches, large steps at large ones) wastes less, at the cost of capture time (a few hundred milliseconds per graph) and the graphs' memory; later versions are configured exactly that way.

2. Where the coefficient goes. Suppose a batch of requests all finish within 2000 steps and are never retracted. What is new_token_ratio's trajectory? And with a retraction every 100 steps?

A way to approach it

From 0.7 by 0.001 per step, it reaches 0.2 after 700 steps (the floor being no lower than the value scaled from base_min_new_token_ratio) and stays there. Frequent retractions keep lifting it back up (new_token_ratio_recovery), so the system oscillates between admitting more and retracting and the throughput falls — which is why the coefficient has to be able both to decay and to recover.

3. Verify the blog's "under 4K lines". From v020-core-lines.sh's result, judge which range the blog means; and what it becomes with memory_pool.py from mem_cache and the attention kernels from layers/ added.

A way to approach it

The controller directory is about 3000 lines, which fits "under 4K"; adding memory_pool.py (about 100 lines) and the three Triton attention files (about 900) is still around 4K. The claim holds, but it excludes the model files and the service layer.

How to explain it

"Why is SGLang fast?" — Do not stop at RadixAttention. Give v0.2's three sources: CUDA graphs kill the launch overhead at small batches; admission by estimate plus retraction keeps the memory full without deadlocking; decoding in a run reduces the round trips into Python. Then add "all of it in 3000 lines of Python", and that zero-overhead scheduling later hid the Python overhead entirely — putting it that way shows you have read the code and not just the blog's headline.

Summary

• v0.2.0 (2024-07-25): up to 3.1 times vLLM 0.5.2's throughput on Llama-70B, with a scheduling core of about 3000 lines.
• Three sources: CUDA graphs by default (fixed buffers plus padding plus a shared memory pool), a dynamic new_token_ratio plus retraction, and the balance between decoding in a run and streaming.
• The fortnight around the release: bench_serving, DeepSeek-V2, chunked prefill and the directory restructuring — the next phase's seeds are all here.

Found a mistake or a better way to say it? Edit this page on GitHub · Report an issue