mini-sglang from Scratch Stage 2 · Scheduling it well: many requests at once

The scheduler skeleton and the offline interface

In the last chapter we played the scheduler by hand: allocating KV locations, filling in the batch fields, appending tokens. This chapter automates it. mini-sglang's scheduler is four small managers, for the request table, the KV cache, the decode set and the prefill queue, and the main loop is barely a dozen lines: receive, pick a batch, prepare, forward, handle the results. We start with the plainest version, with no prefix reuse, no chunking and no overlap, and drive it from the offline LLM interface.

Self-test: if you can answer these, skip the chapter

1. What is the difference between continuous batching and static batching?
2. Does a mini-sglang step do only prefill or only decode, or does it mix them? Which comes first?
3. The decode set is a set. Why sort by uid when assembling a batch?
4. The offline LLM interface has no ZMQ. How does it reuse the scheduler's main loop?

Answers (try it yourself first, then expand)

1. Static batching waits for a whole batch of requests to finish before starting the next; continuous batching reassembles the batch every step, so a finished request leaves at once and a new one joins at any time.
2. One kind per step: if there is a prefill that can be admitted it goes first, otherwise it decodes.
3. Under tensor parallelism every rank runs its own scheduler and they must assemble exactly the same batch, because the order of requests decides the KV layout. A set iterates in an unspecified order, so sorting by uid keeps the ranks in agreement.
4. LLM subclasses the scheduler and overrides the two methods for receiving messages and sending results: it takes inputs from an in-memory request list and collects results locally, reusing the main loop unchanged.

Files you will write: scheduler/config.py, scheduler/utils.py, scheduler/decode.py, scheduler/prefill.py, scheduler/scheduler.py, llm/llm.py (scheduler/table.py came in the last chapter; scheduler/cache.py comes in the next one, and this chapter uses its naive mode).

Files so far: 61 (9 new, 0 modified, 7 brought in early)

minisgl/
├── __init__.py
├── core.py
├── env.py  ← new in this chapter
├── attention/  (4 files, unchanged)
├── distributed/  (3 files, unchanged)
├── engine/  (5 files, unchanged)
├── kernel/  (2 files, unchanged)
├── kvcache/  (4 files, unchanged)
├── layers/  (8 files, unchanged)
├── llm/
│   ├── __init__.py  ← new in this chapter
│   ├── llm.py  ← new in this chapter
├── message/
│   ├── __init__.py  ← brought in early (explained in chapter 12)
│   ├── backend.py  ← brought in early (explained in chapter 12)
│   ├── utils.py  ← brought in early (explained in chapter 12)
│   ├── tokenizer.py  ← brought in early (explained in chapter 12)
├── models/  (7 files, unchanged)
├── moe/  (2 files, unchanged)
├── scheduler/
│   ├── table.py
│   ├── __init__.py  ← new in this chapter
│   ├── config.py  ← new in this chapter
│   ├── utils.py  ← new in this chapter
│   ├── decode.py  ← new in this chapter
│   ├── prefill.py  ← new in this chapter
│   ├── scheduler.py  ← new in this chapter
│   ├── cache.py  ← brought in early (explained in chapter 8)
│   ├── io.py  ← brought in early (explained in chapter 14)
├── utils/
│   ├── __init__.py
│   ├── misc.py
│   ├── hf.py
│   ├── torch_utils.py
│   ├── registry.py
│   ├── logger.py
│   ├── device.py
│   ├── mp.py  ← brought in early (explained in chapter 12)

This step's main: examples/ch07_scheduler.py — it uses only the files above; python tools/steps.py check rebuilds this tree chapter by chapter and runs it.

Animation: continuous batching and admission control (about 1.5 minutes, covering this chapter and the next; Chinese narration and subtitles)

The parts

minisgl/scheduler/scheduler.py · Scheduler.__init__

def __init__(self, config: SchedulerConfig):
    from minisgl.engine import Engine

    self.engine = Engine(config)
    self.config = self.engine.config
    # the scheduler prepares metadata on its own stream while the engine computes on another, so the two overlap
    self.device = self.engine.device
    self.stream = create_stream(self.device)
    self.engine_stream_ctx = stream_ctx(self.engine.stream)
    set_stream(self.stream)

    self.table_manager = TableManager(config.max_running_req, self.engine.page_table)
    self.cache_manager = CacheManager(
        self.engine.num_pages, config.page_size, self.engine.page_table, config.cache_type)
    self.decode_manager = DecodeManager(config.page_size)
    self.prefill_manager = PrefillManager(
        self.cache_manager, self.table_manager, self.decode_manager)

    self.finished_reqs: Set[Req] = set()
    # under overlap scheduling a released slot can only be reallocated once any batch that might use it is done (see _free_req_resources)
    self._defer_slot_free = False
    self._deferred_slots: List[int] = []
    self.tokenizer = load_tokenizer(config.model_path)
    self.eos_token_id = self.tokenizer.eos_token_id
    self.token_pool = self.table_manager.token_pool
    self.prefill_budget = config.max_extend_tokens
    super().__init__(config, self.engine.tp_cpu_group)

what the scheduler is made of, and its main loop

Manager	Responsibility	Chapter
TableManager	rows of the page table / token pool: one per running request	KV pool
CacheManager	page allocation and release in the KV pool, the face of the prefix cache	CacheManager
DecodeManager	the set of requests currently decoding	this chapter
PrefillManager	the queue waiting for prefill, and assembling prefill batches	this chapter, chunking in chapter 10

The scheduler also inherits SchedulerIOMixin, which provides receive_msg and send_result; in an online service they go over ZMQ (chapter 14), and offline LLM overrides them.

The main loop

minisgl/scheduler/scheduler.py · Scheduler.normal_loop

def normal_loop(self) -> None:
    """The plain loop: receive, schedule, forward, wait, handle results. The CPU and the GPU take turns."""
    self._defer_slot_free = False  # the plain loop has no in-flight batch, so the release takes effect at once
    blocking = not (self.prefill_manager.runnable or self.decode_manager.runnable)
    for msg in self.receive_msg(blocking=blocking):
        self._process_one_msg(msg)
    forward_input = self._schedule_next_batch()
    ongoing = None
    if forward_input is not None:
        ongoing = (forward_input, self._forward(forward_input))
    self._process_last_data(ongoing)

Four steps per iteration:

1. Receive. If there is nothing to do at all (nothing waiting for prefill, nothing decoding), block and wait; otherwise take only the messages that have already arrived, without waiting.

2. Pick a batch. Prefill if anything is waiting, decode otherwise:

   minisgl/scheduler/scheduler.py · Scheduler._schedule_next_batch

   def _schedule_next_batch(self) -> ForwardInput | None:
       # prefill first: prefill whenever anything is waiting, decode otherwise
       batch = (self.prefill_manager.schedule_next_batch(self.prefill_budget)
                or self.decode_manager.schedule_next_batch())
       return self._prepare_batch(batch) if batch else None
   

   This is the "prefill first" policy: new requests get their first token as soon as possible (low TTFT), at the cost of interrupting the decoding requests for a step. One kind per step, never mixed, which keeps each step's batch shape simple (a decode batch can use CUDA Graph) and is one of the places mini-sglang is simpler than production SGLang. 3. Prepare and run forward. _prepare_batch fills in the batch's fields and _forward calls the engine. 4. Handle the results. Give the new token to the request, decide whether it is finished, reply to the detokenizer, and release the resources of whatever finished.

run_forever just calls this loop over and over (the overlap version, overlap_loop, arrives in chapter 11).

Two of the managers

PrefillManager keeps a first-come-first-served waiting queue and uses a PrefillAdder each step to admit as many requests as it can, starting from the head:

minisgl/scheduler/prefill.py · PrefillManager.schedule_next_batch

def schedule_next_batch(self, prefill_budget: int) -> Batch | None:
    if not self.pending_list:
        return None
    adder = PrefillAdder(
        token_budget=prefill_budget,
        reserved_size=self.decode_manager.inflight_tokens,  # the space already promised to the decoding requests
        cache_manager=self.cache_manager,
        table_manager=self.table_manager,
    )
    reqs: List[Req] = []
    chunked_list: List[PendingReq] = []
    for pending_req in self.pending_list:  # first come first served: stop at the first that does not fit
        req = adder.try_add_one(pending_req)
        if req is None:
            break
        pending_req.chunked_req = None
        if isinstance(req, ChunkedReq):
            pending_req.chunked_req = req
            chunked_list.append(pending_req)
        reqs.append(req)
    if not reqs:
        return None
    # an unfinished chunked request goes to the head of the queue and continues first next step
    self.pending_list = chunked_list + self.pending_list[len(reqs):]
    return Batch(reqs=reqs, phase="prefill")

PrefillAdder does three things per request: match the cached part in the prefix cache, check that there is a free request slot and enough KV space (admission control, covered next chapter), and write this step's tokens into the token pool. It stops at the first request that does not fit, rather than skipping it to admit smaller ones behind it, which keeps things fair and stops any request from being jumped over forever.

DecodeManager is simpler still, just a set of requests:

minisgl/scheduler/decode.py · DecodeManager

@dataclass
class DecodeManager:
    page_size: int
    running_reqs: Set[Req] = field(default_factory=set)

    def filter_reqs(self, reqs: Iterable[Req]) -> None:
        """After a forward pass: merge this step's requests into the running set and drop the ones that can no longer generate."""
        self.running_reqs = {req for req in self.running_reqs.union(reqs) if req.can_decode}

    def remove_req(self, req: Req) -> None:
        self.running_reqs.discard(req)

    def abort_req(self, uid: int) -> Req | None:
        for req in self.running_reqs:
            if req.uid == uid:
                self.running_reqs.remove(req)
                return req
        return None

    @property
    def inflight_tokens(self) -> int:
        """How many more tokens of KV the running requests may still take (plus one page of slack each)."""
        reserved = (self.page_size - 1) * len(self.running_reqs)
        return sum(req.remain_len for req in self.running_reqs) + reserved

    def schedule_next_batch(self) -> Batch | None:
        if not self.runnable:
            return None
        # sort by uid: the set may iterate differently on each TP rank, and sorting keeps everyone assembling the same batch
        return Batch(reqs=sorted(self.running_reqs, key=lambda r: r.uid), phase="decode")

    @property
    def runnable(self) -> bool:
        return len(self.running_reqs) > 0

After each forward pass, filter_reqs merges this step's requests in and drops the ones that can no longer generate (can_decode false). A request that has just prefilled joins the decode set this way, naturally. Batches sort by uid: under tensor parallelism every rank has its own DecodeManager, and Python's set iterates in an order that depends on object addresses, which can differ between processes; yet every rank's batch must be identical, the same requests in the same order, or an all-reduce adds different requests' data together. The most recent commit in the official repository fixes exactly this.

Preparing a batch

minisgl/scheduler/scheduler.py · Scheduler._prepare_batch

def _prepare_batch(self, batch: Batch) -> ForwardInput:
    self.engine.graph_runner.pad_batch(batch)
    self.cache_manager.allocate_paged(batch.reqs)
    batch.positions = _make_positions(batch, self.device)
    input_mapping = _make_input_tuple(batch, self.device)
    write_mapping = _make_write_tuple(batch, self.device)
    batch.out_loc = self.engine.page_table[input_mapping]
    self.engine.attn_backend.prepare_metadata(batch)
    return ForwardInput(batch, self.engine.sampler.prepare(batch), input_mapping, write_mapping)

In order:

1. Pad (chapter 18; a no-op without CUDA Graph);
2. Allocate KV pages: find locations for this step's tokens and write them into the page table (chapter 8);
3. Positions: [cached_len, device_len) per request, packed into one dimension;
4. Two pairs of two-dimensional indices: input_tuple = (row, position) to take the input from the token pool and out_loc from the page table; write_tuple = (row, device_len) to write the sample back into the token pool's next position;
5. Attention metadata and sampling parameters.

minisgl/scheduler/scheduler.py · _make_write_tuple

def _make_write_tuple(batch: Batch, device: torch.device) -> Indice2D:
    """The sampled token is written to token_pool[table_idx, device_len], the next step's input position.

    A column of -1 means do not write: a chunked request, or one that can no longer generate
    (writing -1 lands in the last column, harmlessly). Note that device_len is incremented inside
    forward_batch, so the value read here is the one before, which is exactly the new token's
    position.
    """
    rows = torch.tensor([r.table_idx for r in batch.reqs], dtype=torch.int64, pin_memory=pin(device))
    cols = torch.tensor([r.device_len if r.can_decode else -1 for r in batch.reqs],
                        dtype=torch.int64, pin_memory=pin(device))
    return rows.to(device, non_blocking=True), cols.to(device, non_blocking=True)

Then the forward pass:

minisgl/scheduler/scheduler.py · Scheduler._forward

def _forward(self, forward_input: ForwardInput) -> ForwardOutput:
    batch, sample_args, input_mapping, write_mapping = forward_input
    batch.input_ids = self.token_pool[input_mapping]  # the input is gathered on the device, never through the CPU
    output = self.engine.forward_batch(batch, sample_args)
    self.token_pool[write_mapping] = output.next_tokens_gpu  # the sample is written straight back as the next step's input
    self.decode_manager.filter_reqs(batch.reqs)
    return output

Both the input token_pool[input_tuple] and the write-back token_pool[write_tuple] = next_tokens are indexing operations on the device, so the CPU never needs to know the token values.

Handling the results

minisgl/scheduler/scheduler.py · Scheduler._process_last_data

def _process_last_data(self, last_data: ForwardData | None) -> None:
    if last_data is None:
        return
    batch, (_, next_tokens_cpu, copy_done) = last_data[0].batch, last_data[1]
    copy_done.synchronize()  # wait for the sample to reach the CPU, which also means this batch and every earlier one are done
    self._release_deferred_slots()
    reply: List[DetokenizeMsg] = []
    new_finished: Set[Req] = set()
    with self.cache_manager.lazy_free_region():
        for i, req in enumerate(batch.reqs):
            if isinstance(req, ChunkedReq):  # a chunked request produces no output this step
                continue
            if req in self.finished_reqs:
                # under overlap scheduling the request finished last step (it hit EOS) but that was not handled yet, so this step had already
                # it was already included. This step's result is redundant and is dropped (upstream sends one extra message, which the frontend happens to ignore).
                continue
            next_token = next_tokens_cpu[i]
            req.append_host(next_token.unsqueeze(0))
            token = int(next_token.item())
            # req.can_decode cannot decide whether max_tokens was reached: under overlap scheduling, by the time step N's results are handled
            # step N+1 has already been launched and req's device_len has moved on, so it would mark finished one token early.
            # judge by the number of tokens the CPU has received, which is independent of the scheduling rhythm.
            finished = len(req.input_ids) >= req.max_device_len
            if not req.sampling_params.ignore_eos:
                finished |= token == self.eos_token_id
            reply.append(DetokenizeMsg(uid=req.uid, next_token=token, finished=finished))
            if finished:
                self.decode_manager.remove_req(req)
                self._free_req_resources(req)
                new_finished.add(req)
            elif batch.is_prefill:  # prefill has just finished: hand the prompt's KV to the prefix cache right away
                self.cache_manager.cache_req(req, finished=False)
    self.finished_reqs = new_finished
    self.send_result(reply)

For each request in the batch: append the new token to the CPU-side input_ids, decide whether it has finished (it reached max_tokens, or hit EOS without ignore_eos), and build one DetokenizeMsg. A finished request releases its resources: it returns its page-table row and hands its KV to the cache (the naive cache frees all of it). A request that has just prefilled hands the prompt's KV to the prefix cache (chapter 9).

Two things in this code are there for overlap scheduling in chapter 11, namely finished_reqs and using len(req.input_ids) rather than req.can_decode to judge the length, and that chapter explains them.

Handling a new request:

minisgl/scheduler/scheduler.py · Scheduler._process_one_msg

def _process_one_msg(self, msg: BaseBackendMsg) -> None:
    if isinstance(msg, BatchBackendMsg):
        for m in msg.data:
            self._process_one_msg(m)
    elif isinstance(msg, ExitMsg):
        raise KeyboardInterrupt
    elif isinstance(msg, UserMsg):
        input_len, max_seq_len = len(msg.input_ids), self.engine.max_seq_len
        max_output_len = max_seq_len - input_len
        if max_output_len <= 0:
            logger.warning_rank0(f"Input length {input_len} exceeds {max_seq_len}, "
                                 f"request {msg.uid} dropped.")
            return
        if msg.sampling_params.max_tokens > max_output_len:
            msg.sampling_params.max_tokens = max_output_len
        self.prefill_manager.add_one_req(msg)
    elif isinstance(msg, AbortBackendMsg):
        req = self.prefill_manager.abort_req(msg.uid) or self.decode_manager.abort_req(msg.uid)
        if req is not None:
            self._free_req_resources(req)
            # under overlap scheduling it may still be in an in-flight batch: marking it finished makes that step's result be discarded,
            # cache_req is never called again on an already released request
            self.finished_reqs.add(req)
    else:
        raise NotImplementedError(f"Unknown message type: {type(msg)}")

When the prompt plus max_tokens exceeds the model's maximum length, max_tokens is cut back; a request whose prompt alone is too long is dropped outright (upstream likewise only logs a warning, and the frontend never gets a reply, which is one thing worth improving).

The offline LLM interface

minisgl/llm/llm.py · LLM

class LLM(Scheduler):
    """The offline inference interface: no processes at all, reusing the scheduler's main loop directly.

    It overrides the scheduler's two IO methods: receive_msg takes requests from the pending list
    (feeding them in within the prefill budget) and send_result records the generated tokens. Once
    everything is done and the scheduler would block for a new message, RequestAllFinished breaks
    out of run_forever.
    """

    def __init__(self, model_path: str, dtype: torch.dtype = torch.bfloat16, **kwargs):
        config = SchedulerConfig(model_path=model_path, tp_info=DistributedInfo(0, 1),
                                 dtype=dtype, offline_mode=True, **kwargs)
        super().__init__(config)
        self.pending_requests: List[Tuple[List[int] | str, SamplingParams]] = []
        self.status_map: Dict[int, RequestStatus] = {}
        self.counter = 0

    def _tokenize_one(self, prompt: List[int] | str) -> torch.Tensor:
        if isinstance(prompt, str):
            return self.tokenizer.encode(prompt, return_tensors="pt").view(-1).to(torch.int32)
        return torch.tensor(prompt, dtype=torch.int32)

    def offline_receive_msg(self, blocking: bool = False) -> List[BaseBackendMsg]:
        if blocking and not self.pending_requests:
            raise RequestAllFinished()
        results: List[BaseBackendMsg] = []
        added = total_len = 0
        for prompt, sp in self.pending_requests:
            if total_len >= self.prefill_budget:
                break
            input_ids = self._tokenize_one(prompt)
            total_len += len(input_ids)
            uid = self.counter + added
            added += 1
            results.append(UserMsg(uid=uid, input_ids=input_ids, sampling_params=sp))
            self.status_map[uid] = RequestStatus(uid, input_ids.tolist(), [])
        self.counter += added
        self.pending_requests = self.pending_requests[added:]
        return results

    def offline_send_result(self, reply: List[DetokenizeMsg]) -> None:
        for msg in reply:
            status = self.status_map[msg.uid]
            if not (msg.finished and msg.next_token == self.eos_token_id):
                status.output_ids.append(msg.next_token)

    def generate(self, prompts: List[str] | List[List[int]],
                 sampling_params: List[SamplingParams] | SamplingParams
                 ) -> List[Dict[str, str | List[int]]]:
        self.pending_requests, self.status_map, self.counter = [], {}, 0
        if isinstance(sampling_params, SamplingParams):
            sampling_params = [sampling_params] * len(prompts)
        # copy it every time: the scheduler may change max_tokens
        self.pending_requests = [(p, SamplingParams(**vars(sp)))
                                 for p, sp in zip(prompts, sampling_params)]
        try:
            self.run_forever()
        except RequestAllFinished:
            pass
        return [{"text": self.tokenizer.decode(self.status_map[i].output_ids),
                 "token_ids": self.status_map[i].output_ids} for i in range(len(prompts))]

LLM subclasses Scheduler, passes offline_mode=True so the IO layer creates no ZMQ queues, and overrides two methods:

• offline_receive_msg: take requests from the pending list and feed them in batches within the prefill budget (max_extend_tokens). When the scheduler has nothing to do and would block waiting for a message while the list is also empty, everything is done and it raises RequestAllFinished to break out of run_forever;
• offline_send_result: record each reply's token on the matching request (EOS is not recorded).

This small trick lets the offline interface share the main loop with the online service, so testing one tests the other's scheduling logic too.

Running it

examples/ch07_scheduler.py

"""Chapter 7: the offline interface running 5 requests, printing the batch scheduled each step.

To see only the skeleton, overlap scheduling is off and the naive cache is used (no prefix reuse).
"""

import torch
from minisgl.core import SamplingParams
from minisgl.env import ENV
from minisgl.llm import LLM
from transformers import AutoModelForCausalLM, AutoTokenizer

ENV.DISABLE_OVERLAP_SCHEDULING.value = True
path = "models/Qwen3-0.6B"
llm = LLM(path, dtype=torch.float32, max_running_req=8, num_page_override=1024,
          max_seq_len_override=512, cache_type="naive")
orig = llm._forward


def traced(fi):
    b = fi.batch
    detail = ", ".join(f"uid{r.uid}:{r.extend_len}" for r in b.reqs)
    print(f"  {b.phase:<7} bs={b.size}  tokens={len(b.positions):<3} [{detail}]")
    return orig(fi)


llm._forward = traced
prompts = ["The capital of France is", "List three prime numbers:", "def fibonacci(n):",
           "1 + 1 =", "Once upon a time,"]
lengths = [6, 2, 5, 1, 4]  # the requests have different max_tokens: the short ones finish and leave first
print("the scheduling trace:")
outputs = llm.generate(prompts, [SamplingParams(max_tokens=n, ignore_eos=True) for n in lengths])
tok = AutoTokenizer.from_pretrained(path)
hf = AutoModelForCausalLM.from_pretrained(path, dtype=torch.float32)
for p, n, o in zip(prompts, lengths, outputs):
    ids = tok(p, return_tensors="pt").input_ids
    ref = hf.generate(ids, max_new_tokens=n, min_new_tokens=n, do_sample=False)[0, ids.shape[1]:].tolist()
    print(f"{p!r:<30} -> {o['text']!r:<32} matches HF: {o['token_ids'] == ref}")
llm.shutdown()

What it prints

the scheduling trace:
  prefill bs=5  tokens=24  [uid0:5, uid1:5, uid2:4, uid3:5, uid4:5]
  decode  bs=4  tokens=4   [uid0:1, uid1:1, uid2:1, uid4:1]
  decode  bs=3  tokens=3   [uid0:1, uid2:1, uid4:1]
  decode  bs=3  tokens=3   [uid0:1, uid2:1, uid4:1]
  decode  bs=2  tokens=2   [uid0:1, uid2:1]
  decode  bs=1  tokens=1   [uid0:1]
'The capital of France is'     -> ' Paris. The capital of Italy'   matches HF: True
'List three prime numbers:'    -> ' 2'                             matches HF: True
'def fibonacci(n):'            -> '\n    if n =='                  matches HF: True
'1 + 1 ='                      -> ' '                              matches HF: True
'Once upon a time,'            -> ' there were 3'                  matches HF: True

The first step prefills all five requests together (24 tokens packed into one dimension). uid3, with max_tokens=1, finishes right after prefill and never reaches decode; uid1 leaves after one decode step; the rest carry on and the batch shrinks. That is continuous batching: requests join and leave at their own pace, without waiting for the whole batch. All five outputs match what Hugging Face generates for them individually.

The official implementation

• the main loop: scheduler/scheduler.py · Scheduler.normal_loop (lines 108-118)
• preparing and running forward: scheduler/scheduler.py · Scheduler._prepare_batch (lines 204-217), scheduler/scheduler.py · Scheduler._forward (lines 227-233)
• the prefill queue: scheduler/prefill.py · PrefillManager.schedule_next_batch (lines 126-151)
• the decode set: scheduler/decode.py · DecodeManager (lines 9-39)
• the offline interface: llm/llm.py · LLM (lines 28-98)

Tests

tests/test_ch07_scheduler.py · test_requests_with_different_lengths_leave_and_join

def test_requests_with_different_lengths_leave_and_join():
    """Different max_tokens: the short ones finish and leave the batch first while the rest carry on, and a later generate's requests join afterwards."""
    ENV.DISABLE_OVERLAP_SCHEDULING.value = True
    llm = build_llm(QWEN3, cache_type="naive")
    lengths = [3, 12, 7, 1, 9]
    out = llm.generate(PROMPTS, [greedy(n) for n in lengths])
    ref = hf_greedy(QWEN3, PROMPTS, 12)
    assert [o["token_ids"] for o in out] == [r[:n] for r, n in zip(ref, lengths)]
    assert llm.table_manager.available_size == 8  # every row has been returned
    llm.shutdown()

How to explain it

On the scheduler: continuous batching reassembles the batch step by step, so requests join at any time and leave the moment they are done, without waiting for the whole batch (in static batching a short request has to sit through the longest one). mini-sglang's main loop is receive, pick a batch, prepare, forward, handle results, and it does one kind per step: prefill first (first come first served, stopping at the first that does not fit), decode otherwise. The decode set sorts by uid when assembling, so every TP rank builds exactly the same batch. Against vLLM V1: vLLM mixes prefill and decode in one step under a unified token budget, while mini-sglang keeps them apart, which is simpler to implement but lets a long prefill stall decode for a step. The offline LLM interface replaces only the receive and send methods and reuses the same main loop.

Exercises

1. Change the policy to "decode first": decode whenever anything is decoding, and prefill only when nothing is. What goes wrong? (Hint: think about when a new request gets served in a system that always has something decoding.)
2. PrefillManager stops at the first request that does not fit. What happens to throughput if it skips that one and keeps trying the ones behind it? What is the risk?
3. In _process_one_msg an over-long prompt is dropped and the frontend never hears back. How would you return an error instead? Which message types need changing?

Answers

1. As long as the decode set is non-empty, a new request never gets prefilled, so TTFT is unbounded (starvation). A practical compromise is to cap how many decode steps may run in a row, or to mix prefill and decode in one batch the way production SGLang does.
2. Throughput may go up, since small requests fill the budget, but a large request can be jumped over by later small ones forever and starve. That needs an aging mechanism, where waiting longer raises priority.
3. Have the scheduler reply with a DetokenizeMsg carrying an error flag (an added error field with finished=True), have the detokenizer turn it into a UserReply with the error message, and have the API server return a 400 on that basis.

Summary

• The scheduler is four managers plus a main loop: receive, pick a batch (prefill first, one kind per step), prepare, forward, handle results.
• The prefill queue is first come first served and stops at the first request that does not fit; the decode set is updated after each forward pass and sorts by uid so every rank agrees.
• Inputs are taken from the token pool and samples written back to it, all on the device.
• The offline LLM interface overrides the two IO methods and reuses the same main loop.

Exercises for this chapter (1)

Write the code and run the tests in your browser, or judge locally with python practice/judge.py. The exercises are in Chinese for now.

• 191. PrefillAdder：准入控制与分块 prefill (medium)

Found a mistake or a better way to say it? Edit this page on GitHub · Report an issue