Skip to content
mini-sglang from Scratch Stage 3 · Serving: make it a service

The tokenizer and incremental detokenization

The tokenizer process does two opposite things: turn the text (or conversation) the API server sends into tokens for the scheduler, and turn the new tokens the scheduler sends back into text for the API server. The first is one apply_chat_template and one encode; the second is much harder, because a Chinese character or an emoji can be split across two or three tokens, decoding one alone gives nothing but "�", and decoding is not additive. This chapter implements the tokenizer process, with the focus on incremental detokenization.

Self-test: if you can answer these, skip the chapter

  1. Why not call tokenizer.decode([token]) on each new token and send that to the frontend?
  2. Why is detokenization "not additive"? Give an example where decode(a + b) != decode(a) + decode(b).
  3. Why can the tokenizer and the detokenizer share one process? When should they be separated?
  4. Should the EOS token be sent to the frontend?
Answers (try it yourself first, then expand)
  1. One token may be only part of a character (the UTF-8 bytes of Chinese text or an emoji get split), and decoding it alone gives mojibake; the spaces and merge rules between neighbouring tokens also make piecewise decoding differ from decoding the whole.
  2. With byte-level BPE the 3 UTF-8 bytes of a Chinese character can land in two tokens: decoded separately they give two pieces of mojibake (replacement characters), and only together do they give the right character. Likewise SentencePiece's leading-space marker gets stripped when a token is decoded alone.
  3. Tokenizing and detokenizing are both light CPU work, so one process is enough and it saves a hop between processes; when there are many requests and tokenizing or detokenizing becomes the bottleneck (long prompts, high concurrency), split it into several processes (but the detokenizer's state is per-process, so a split has to route by request).
  4. No: EOS only marks the end of generation, it is not text to display. Stop on it server-side and return it as the finish_reason.

Files you will write: tokenizer/tokenize.py, tokenizer/detokenize.py, tokenizer/server.py.

Files so far: 67 (4 new, 0 modified, 0 brought in early)
minisgl/
├── __init__.py
├── core.py
├── env.py
├── attention/  (4 files, unchanged)
├── distributed/  (3 files, unchanged)
├── engine/  (5 files, unchanged)
├── kernel/  (2 files, unchanged)
├── kvcache/  (5 files, unchanged)
├── layers/  (8 files, unchanged)
├── llm/  (2 files, unchanged)
├── message/  (5 files, unchanged)
├── models/  (7 files, unchanged)
├── moe/  (2 files, unchanged)
├── scheduler/  (9 files, unchanged)
├── tokenizer/
│   ├── __init__.py  ← new in this chapter
│   ├── tokenize.py  ← new in this chapter
│   ├── detokenize.py  ← new in this chapter
│   ├── server.py  ← new in this chapter
├── utils/  (8 files, unchanged)
This step's main: examples/ch13_tokenizer.py — it uses only the files above; python tools/steps.py check rebuilds this tree chapter by chapter and runs it.

Tokenizing

minisgl/tokenizer/tokenize.py · TokenizeManager
class TokenizeManager:
    def __init__(self, tokenizer: PreTrainedTokenizerBase) -> None:
        self.tokenizer = tokenizer

    def tokenize(self, msgs: List[TokenizeMsg]) -> List[torch.Tensor]:
        results: List[torch.Tensor] = []
        for msg in msgs:
            if isinstance(msg.text, list):  # a list of chat messages: apply the chat template first
                prompt = self.tokenizer.apply_chat_template(
                    msg.text, tokenize=False, add_generation_prompt=True)
            else:
                prompt = msg.text
            ids = self.tokenizer.encode(prompt, return_tensors="pt")
            results.append(ids.view(-1).to(torch.int32))
        return results
TokenizeMsg.text can be plain text (the /generate endpoint, or prompt in an OpenAI request) or a list of conversation messages (OpenAI's messages). The latter is expanded into text by the model's own chat template, with the "the assistant starts answering" prompt added (add_generation_prompt=True). Getting the template wrong is one of the most common quality problems in an inference service: the model continues the user's sentence, or never stops (see the tokenization chapter of the LLM Internals handbook).

Incremental detokenization

A byte-level BPE vocabulary is built over UTF-8 bytes: an uncommon Chinese character (3 bytes) or an emoji (4 bytes) is very likely split across two tokens. Decoding the first one alone yields incomplete bytes, shown as the replacement character "�". SentencePiece-style tokenizers also decide from context whether to add a leading space, so decoding one by one and concatenating can differ from decoding the whole.

The usual approach, shared by SGLang, vLLM and HF's TextStreamer, is to decode only a recent window and take the difference between two decodes as the new text:

minisgl/tokenizer/detokenize.py · DetokenizeManager.detokenize
def detokenize(self, msgs: List[DetokenizeMsg]) -> List[str]:
    read_ids: List[List[int]] = []
    surr_ids: List[List[int]] = []
    for msg in msgs:
        s = self.decode_map.setdefault(msg.uid, DecodeStatus([], "", 0, 0, 0))
        if not (msg.finished and msg.next_token == self.eos_token_id):  # EOS is not emitted
            s.decoded_ids.append(msg.next_token)
        read_ids.append(s.decoded_ids[s.surr_offset :])
        surr_ids.append(s.decoded_ids[s.surr_offset : s.read_offset])
    # decoding a group together beats one call each
    read_texts = self.tokenizer.batch_decode(read_ids)
    surr_texts = self.tokenizer.batch_decode(surr_ids)

    outputs: List[str] = []
    for msg, read_str, surr_str in zip(msgs, read_texts, surr_texts, strict=True):
        s = self.decode_map[msg.uid]
        new_text = read_str[len(surr_str) :]
        if new_text and not new_text.endswith("�"):  # the new text is complete: move the window forward
            output_str = s.decoded_str + new_text
            s.decoded_str = output_str
            s.surr_offset = s.read_offset
            s.read_offset = len(s.decoded_ids)
        else:  # incomplete: emit only what is certain and leave the window put
            output_str = s.decoded_str + find_printable_text(new_text)
        outputs.append(output_str[s.sent_offset :])
        s.sent_offset = len(output_str)
        if msg.finished:
            del self.decode_map[msg.uid]
    return outputs
Each request keeps a few offsets:

  • the tokens between surr_offset and read_offset are already confirmed output, kept to provide context;
  • everything after read_offset is new and unconfirmed.

Each time it decodes both [surr_offset, end] and [surr_offset, read_offset], and the first minus the second is the new text. If the new text is complete (it does not end in "�"), it is confirmed and the window moves forward; otherwise the window stays put and only the certain part is emitted (find_printable_text: up to a newline, up to a Chinese character, or up to the last space), waiting for later tokens to complete the character. sent_offset records how many characters have gone to the frontend, so each character is sent exactly once.

examples/ch13_tokenizer.py
"""Chapter 13: the chat template, and how incremental detokenization avoids emitting half a character."""

from minisgl.core import SamplingParams
from minisgl.message import DetokenizeMsg, TokenizeMsg
from minisgl.tokenizer import DetokenizeManager, TokenizeManager
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("models/Qwen3-0.6B")
ids = TokenizeManager(tok).tokenize([TokenizeMsg(
    uid=0, text=[{"role": "user", "content": "Hello"}], sampling_params=SamplingParams())])[0]
print("after the chat template:", repr(tok.decode(ids)))

text = "鱻䨻🦩 is rare, 推理 naïve"  # each of the first three characters is split across two tokens
out_ids = tok(text).input_ids
dm = DetokenizeManager(tok)
print(f"{'token':>7}  {'decoded alone':<16} delta output")
pieces = []
for i, t in enumerate(out_ids):
    piece = dm.detokenize([DetokenizeMsg(uid=1, next_token=t, finished=i == len(out_ids) - 1)])[0]
    pieces.append(piece)
    print(f"{t:>7}  {tok.decode([t])!r:<12} {piece!r}")
print("concatenated:", repr("".join(pieces)), " same as decoding the whole:", "".join(pieces) == tok.decode(out_ids))
What it prints
after the chat template: '<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\n'
  token  decoded alone    delta output
 100024  '�'          ''
    119  '�'          '鱻'
 128237  '�'          ''
    119  '�'          '䨻'
 123918  '�'          ''
    102  '�'          '🦩'
    374  ' is'        ' is'
   8848  ' rare'      ' rare'
     11  ','          ','
  46602  ' �'         ' '
    101  '�'          '推'
  21887  '理'          '理'
  94880  ' naï'       ' naï'
    586  've'         've'
concatenated: '鱻䨻🦩 is rare, 推理 naïve'  same as decoding the whole: True
The character 鱻 is split across two tokens, so the first produces no delta and only the second emits the complete 鱻; 推 shares a token with the space before it, and the first token emits only the space it is sure of. Concatenated, the result is identical to decoding the whole, and the frontend never receives a "�".

Other details:

  • a group of messages is decoded together with batch_decode, which beats one call each;
  • when a request has finished (finished=True) and its last token is EOS, the EOS is left out of the decode (it would come back as a special string like <|im_end|>);
  • a finished request's state is deleted, or a long-running service leaks memory (which is exactly what made the stale messages of "problem two" in the last chapter harmful).

The tokenizer process

minisgl/tokenizer/server.py · tokenize_worker
@torch.inference_mode()
def tokenize_worker(*, tokenizer_path: str, addr: str, create: bool, backend_addr: str,
                    frontend_addr: str, local_bs: int, tokenizer_id: int = -1,
                    ack_queue: mp.Queue | None = None) -> None:
    """The tokenizer process's main loop. One process both tokenizes (frontend to scheduler) and detokenizes (scheduler to frontend)."""
    send_backend = ZmqPushQueue(backend_addr, create=False, encoder=BaseBackendMsg.encoder)
    send_frontend = ZmqPushQueue(frontend_addr, create=False, encoder=BaseFrontendMsg.encoder)
    recv_listener = ZmqPullQueue(addr, create=create, decoder=BatchTokenizerMsg.decoder)
    tokenizer = load_tokenizer(tokenizer_path)
    logger = init_logger(__name__, f"tokenizer_{tokenizer_id}")
    tokenize_manager = TokenizeManager(tokenizer)
    detokenize_manager = DetokenizeManager(tokenizer)
    if ack_queue is not None:
        ack_queue.put(f"Tokenize server {tokenizer_id} is ready")

    try:
        while True:
            pending = _unwrap_msg(recv_listener.get())
            while len(pending) < local_bs and not recv_listener.empty():
                pending.extend(_unwrap_msg(recv_listener.get()))
            logger.debug(f"Received {len(pending)} messages")
            detok = [m for m in pending if isinstance(m, DetokenizeMsg)]
            tok = [m for m in pending if isinstance(m, TokenizeMsg)]
            abort = [m for m in pending if isinstance(m, AbortMsg)]
            if detok:
                texts = detokenize_manager.detokenize(detok)
                replies = [UserReply(uid=m.uid, incremental_output=t, finished=m.finished)
                           for m, t in zip(detok, texts, strict=True)]
                send_frontend.put(_pack(replies, BatchFrontendMsg))
            if tok:
                tensors = tokenize_manager.tokenize(tok)
                reqs = [UserMsg(uid=m.uid, input_ids=t, sampling_params=m.sampling_params)
                        for m, t in zip(tok, tensors, strict=True)]
                send_backend.put(_pack(reqs, BatchBackendMsg))
            if abort:
                send_backend.put(_pack([AbortBackendMsg(uid=m.uid) for m in abort],
                                       BatchBackendMsg))
    except KeyboardInterrupt:
        pass
The main loop: block for one message, then take as many already-arrived messages as it can (up to local_bs), and handle them in three groups by type. A DetokenizeMsg is detokenized and sent to the API server, a TokenizeMsg is tokenized and sent to the scheduler, and an AbortMsg becomes an AbortBackendMsg forwarded to the scheduler. Several results are packed into one Batch...Msg to send.

By default (--num-tokenizer 0) there is one process doing both directions: the TokenizeMsgs from the API server and the DetokenizeMsgs from the scheduler land in one queue. Both are fast, and one process is enough to sustain one GPU's throughput. With many requests and long prompts, --num-tokenizer N starts N dedicated tokenizing processes, separate from the detokenizer.

The official implementation

Tests

tests/test_ch13_tokenizer.py · test_incremental_detokenize_equals_full_decode
def test_incremental_detokenize_equals_full_decode():
    tok = AutoTokenizer.from_pretrained(QWEN3)
    text = "推理引擎 😀 streams text: naïve café, 1+1=2.\n第二行结束。"
    ids = tok(text).input_ids
    dm = DetokenizeManager(tok)
    pieces = []
    for i, t in enumerate(ids):
        pieces += dm.detokenize([DetokenizeMsg(uid=0, next_token=t, finished=i == len(ids) - 1)])
    assert "".join(pieces) == tok.decode(ids)
    assert all("�" not in p for p in pieces)  # an incomplete character never reaches the frontend
    assert 0 not in dm.decode_map  # clear the state once it finishes

How to explain it

On detokenization: you cannot decode([token]) each token as it arrives, because detokenization is not additive. A Chinese character or an emoji may be several byte-level tokens (decoded alone they are mojibake), and spacing and merge rules depend on context. Incremental detokenization decodes only a recent window and takes the difference between two decodes as the new text, holding back when the end is an incomplete character (�) and emitting only what is certain. A conversation goes through the model's own chat template before encoding, and EOS is never sent to the frontend. The tokenizer and the detokenizer share one process by default (both are light); under load tokenizing can be parallelized across processes, but the detokenizer's state is per-process, so there can be only one.

Exercises

  1. When the text does not end in a Chinese character, find_printable_text emits only up to the last space. Why? What does that mean for Japanese and Korean?
  2. What goes wrong if two detokenizer processes exist and a request's messages can land in either?
  3. Add stop-string support to DetokenizeMsg: finish when a given string appears in the output. Should the check live in the detokenizer or in the scheduler?
Answers
  1. An English word may be split across several tokens, and the next token can change how the earlier part decodes (SentencePiece's space handling, for instance), so cutting at a space is the safest. Japanese kana and Korean hangul fall outside _is_chinese_char, so they are treated as "words" and held back until a space or newline, which makes streaming feel worse.
  2. Incremental detokenization's state (decode_map) is per-process. Split one request's tokens across two processes and each sees only part of them, so the output is garbled. There can therefore be only one detokenizer process (or routing fixed by uid).
  3. A stop string lives at the text level and has to be checked after detokenization, so the detokenizer is the natural place; but once it matches, the scheduler still has to be told to abort the request and free its resources, which needs an abort message from the detokenizer to the scheduler. SGLang 0.5.20 instead decodes the last few output tokens inside the scheduler process to check stop strings (Req.check_match_stop_str_prefix and friends in schedule_batch.py).

Summary

  • Tokenizing: a conversation goes through the template before encoding, and the template must be the model's own.
  • Incremental detokenization decodes only a recent window and takes the difference between two decodes as the new text, waiting when a character is incomplete and emitting only what is certain.
  • The tokenizer process handles messages in groups by type, with tokenizing and detokenizing sharing one process by default; the detokenizer's state is per-process, so there can be only one.

Exercises for this chapter (1)

Write the code and run the tests in your browser, or judge locally with python practice/judge.py. The exercises are in Chinese for now.

Found a mistake or a better way to say it? Edit this page on GitHubReport an issue