Skip to content
mini-sglang from Scratch Stage 3 · Serving: make it a service

Stage 3 milestone: make it a service

At the end of stage 2 the engine is still a Python object: you call generate in the same process, hand it every request at once and collect every result at once. The four chapters of stage 3 turn it into a multi-process online service: an API server takes HTTP, a tokenizer process tokenizes, a scheduler process computes, messages travel between them over ZMQ, and the outside sees an OpenAI-compatible interface with streaming and abort on disconnect. This page starts the service first and talks to it with the standard OpenAI client.

What you hold at the end of this stage

  • One command, python -m minisgl --model ..., starts the service; any OpenAI client (or one curl) can connect;
  • Requests arrive one by one, knowing nothing of each other, and still land in the same batch inside the scheduler;
  • Streaming: the first token is sent as soon as it exists, without waiting for the whole answer;
  • When a client disconnects, its request is aborted and its KV pages reclaimed.

Run it first

cd minisgl && python examples/stage3_milestone.py

examples/stage3_milestone.py
"""Stage 3 milestone: stage 2's LLM.generate has become a multi-process OpenAI-compatible service.

Start the service, look at the processes, one non-streaming call, one streaming call (measuring time to
first token), 8 concurrent requests, and finally print a curl line: after this page, any OpenAI client
can talk to the engine you wrote.
"""

import os
import signal
import socket
import subprocess
import sys
import time
from concurrent.futures import ThreadPoolExecutor

import httpx
import psutil
from openai import OpenAI

with socket.socket() as s:
    s.bind(("127.0.0.1", 0))
    port = s.getsockname()[1]
proc = subprocess.Popen(
    [sys.executable, "-m", "minisgl", "--model", "models/Qwen3-0.6B", "--dtype", "float32",
     "--port", str(port), "--num-pages", "4096", "--max-seq-len-override", "1024"],
    stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, start_new_session=True)
try:
    t = time.time()
    while True:                                                   # poll /v1/models until the service is up
        try:
            if httpx.get(f"http://127.0.0.1:{port}/v1/models", timeout=1).status_code == 200:
                break
        except httpx.HTTPError:
            time.sleep(0.5)
    workers = [c for c in psutil.Process(proc.pid).children(recursive=True)
               if "multiprocessing" in " ".join(c.cmdline())]
    print(f"service ready ({time.time() - t:.0f} s): 1 API server process + {len(workers)} worker processes (scheduler, tokenizer), passing messages over ZMQ")
    client = OpenAI(base_url=f"http://127.0.0.1:{port}/v1", api_key="none")

    r = client.chat.completions.create(
        model="qwen3", messages=[{"role": "user", "content": "The capital of France is? Answer in one word. /no_think"}],
        max_tokens=16, temperature=0)
    print(f"non-streaming: {r.choices[0].message.content!r}")

    t0 = time.perf_counter()
    first = None
    chunks = []
    for c in client.chat.completions.create(
            model="qwen3", messages=[{"role": "user", "content": "Count from 1 to 10. /no_think"}],
            max_tokens=32, temperature=0, stream=True):
        if c.choices[0].delta.content:
            first = first or time.perf_counter() - t0
            chunks.append(c.choices[0].delta.content)
    print(f"streaming: {len(chunks)} chunks, the first after {first * 1000:.0f} ms, all of them after {(time.perf_counter() - t0) * 1000:.0f} ms")

    def ask(i: int) -> float:
        t0 = time.perf_counter()
        client.chat.completions.create(model="qwen3", messages=[{"role": "user", "content": f"{i}+{i}=?"}],
                                       max_tokens=16, temperature=0, extra_body={"ignore_eos": True})
        return time.perf_counter() - t0

    t0 = time.perf_counter()
    with ThreadPoolExecutor(8) as ex:
        lat = list(ex.map(ask, range(8)))
    total = time.perf_counter() - t0
    print(f"8 concurrent requests x 16 tokens: {total:.2f} s in total ({8 * 16 / total:.1f} tokens/s), "
          f"each request {min(lat):.2f}-{max(lat):.2f} s; they share one batch, nobody queues")
    print("Try it yourself: curl http://127.0.0.1:PORT/v1/chat/completions -H 'Content-Type: application/json' "
          "-d '{\"model\":\"qwen3\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}'")
finally:
    os.killpg(proc.pid, signal.SIGKILL)
What it prints
service ready (9 s): 1 API server process + 3 worker processes (scheduler, tokenizer), passing messages over ZMQ
non-streaming: '<think>\n\n</think>\n\nParis'
streaming: 32 chunks, the first after 118 ms, all of them after 1027 ms
8 concurrent requests x 16 tokens: 1.35 s in total (94.5 tokens/s), each request 1.34-1.35 s; they share one batch, nobody queues
Try it yourself: curl http://127.0.0.1:PORT/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen3","messages":[{"role":"user","content":"hi"}]}'

Reading the numbers

1 API server process + 3 worker processes. Everything in stage 2 lived in one process; now tokenizing, scheduling and HTTP each have their own, passing messages over ZMQ. The point of multiple processes is not parallel computation — all the computing happens in the scheduler process — but to make sure the HTTP event loop and Python tokenization never stall the step being computed, and vice versa.

Streaming: first chunk after 118 ms, all of it after 1027 ms. Non-streaming returns only once all 32 tokens exist; streaming shows the user the first word after 118 ms. From here on, time to first token (TTFT) and time per output token (TPOT) are two separate metrics, and inference SLAs are written in terms of them.

8 concurrent requests in 1.35 seconds, each taking 1.34–1.35 seconds. Eight requests leave eight threads at once, pass through HTTP, tokenization and ZMQ into the scheduler, and land in the same batch, so each takes almost exactly as long as the others — nobody queues. The effect is the same as handing generate eight prompts in stage 2; the difference is that these came over the network and know nothing of each other.

What one process became

Stage 2 Stage 3 Where
the arguments of generate are the requests message objects + generic serialization + ZMQ queues (PUSH/PULL, PUB/SUB) Messages, serialization and ZMQ
self.tokenizer(prompt) in the same process a separate tokenizer process; detokenization is incremental (one chunk per streamed token) The tokenizer and incremental detokenization
the scheduler receives a Python list the scheduler reads requests from ZMQ and sends results back; with several ranks, rank 0 broadcasts to the rest Scheduler IO and multi-rank sync
no HTTP a FastAPI OpenAI-compatible API, SSE streaming, abort on disconnect; a launcher that starts every process The API server and the launcher

How to read these four chapters

Read messages first: everything the processes of this stage pass to each other is defined in that one file, and the next three chapters only send and receive it. At the API server, really type the curl line this page's script prints last, then set stream=True and look at what each SSE frame contains. After the four chapters, revisit the fixes to the official implementation in chapters 11 and 12 (one stale message sent after EOS, a PUB/SUB that does not wait for subscribers) — all of them sit on this stage's boundaries.

Checkpoint: do these before stage 4

  • Draw the processes and messages a request passes through: HTTP → tokenizer → scheduler → detokenizer → HTTP;
  • Explain why tokenization gets its own process instead of running inside the API server;
  • Count how many serializations one streamed token goes through on its way to a chunk;
  • Interrupt a streaming request with Ctrl-C and confirm its KV pages are reclaimed in the scheduler (chapter 15's test).

Summary

  • Stage 3 turns a single-process generate into a multi-process OpenAI-compatible service with ZMQ messages.
  • Streaming makes time to first token and time per token two separate metrics.
  • Concurrent requests from the network land in one batch naturally: 8 requests at 1.35 seconds each, no queueing.
  • Multiple processes exist so that HTTP and tokenization never stall the computation, not to compute in parallel.

Found a mistake or a better way to say it? Edit this page on GitHubReport an issue