Advanced C++¶
An inference system's foundations are mostly C++: the communication libraries, the KV transfer engine, an operator's host side, the Python extensions, the scheduler's hot path. This handbook is for engineers who have written other languages and only the basics of C++, and aims at being able to read and change that code, and to write systems code that is correct, efficient and survives the sanitizers.
Who this handbook is for¶
You are probably around here:
- you can write a decent program in Python (or Rust, Go, Java) and have written some C / C++ exercises or small tools;
- opening a repository like SGLang's
sgl-kernel, FlashInfer, DeepEP or Mooncake, you follow the gist, but templates, move semantics,std::atomic's memory orders and the various RAII wrappers leave you unsure; - you know C++ "makes memory errors easy", but cannot say which constructs are undefined behaviour, nor how to find them systematically.
Having read this handbook and done the exercises, you should be able to:
- explain object lifetimes and ownership, and write resource management that neither leaks nor dangles and is exception-safe;
- read the template tricks common in inference libraries (compile-time dispatch by data type and head dimension), the RAII handles and the error-handling macros;
- design cache-friendly data layouts and write the memory pools and block allocators that are "an inference system's memory management";
- write correct concurrent code: a thread pool, a blocking queue, a lock-free single-producer single-consumer queue, and prove there is no data race with ThreadSanitizer;
- organize a project with CMake, locate problems with the sanitizers and
perf, and expose an implementation to Python with pybind11 / a PyTorch C++ extension.
How to use it¶
- Take the self-test first. Every chapter opens with one, and if you can answer all of it, skip to the exercises.
- Always compile and run the code. Every program with a file name on it is complete, so copy it and compile it with the command on the page. Compiling with the sanitizers is the default, not an option.
- Do the exercises before reading the answers. Every chapter ends with exercises whose answers are collapsed; the companion C++ problems are graded locally with one command (below).
- Read it against real code. Reading an inference library's C++ maps what came before onto the real constructs in open-source inference libraries, and is worth going back to after each part.
The learning path¶
The chapter-by-chapter path across all the handbooks is in the roadmap, and the week-by-week arrangement for a job-hunting sprint is in the sprint plan. The order within this book:
| Stage | Chapters | What you can do afterwards | Suggested time |
|---|---|---|---|
| 1. Modern C++ essentials | the compilation model and undefined behaviour · value semantics and RAII · move semantics · smart pointers and ownership · templates and constexpr · the standard library from a performance angle | read modern C++ code, write resource management that neither leaks nor dangles | 1 week |
| 2. Memory | object layout, alignment and the cache · allocators and memory pools | design cache-friendly data structures, write a block allocator | 3-4 days |
| 3. Concurrency | threads, locks and condition variables · atomics and memory order · a lock-free queue and a thread pool | write concurrent components that pass TSan | 1 week |
| 4. Engineering and interop | CMake, the sanitizers and perf · pybind11 and PyTorch extensions · reading an inference library · frequent interview questions | build, debug and ship a C++ component on your own and wire it into Python | 3-4 days |
Version conventions¶
- C++20 is the baseline (
-std=c++20), and anything that needs C++23 is marked, like C++23. - Every program with a file name on it is compiled with g++ 12 (
-Wall -Wextra -Werror) and run under AddressSanitizer + UndefinedBehaviorSanitizer by default, with the concurrency chapters' programs run under ThreadSanitizer; the "output" on the page matches the real run line by line. - A program that demonstrates a mistake on purpose says "this program has a bug", and what the page shows is an excerpt of the sanitizer's report.
Before you start: setting up¶
Apple clang on macOS has no LeakSanitizer, so check for leaks with leaks --atExit -- ./a.out. std::jthread needs a recent libc++: brew install llvm and compiling with $(brew --prefix llvm)/bin/clang++ plus -fexperimental-library is the suggestion (tools/check_code.py does this automatically). libc++ differs from Linux's libstdc++ in its implementation details, so output like sizeof(std::string), capacities and hash table bucket counts will not match the page: the output on the page is Linux + GCC. The full Mac setup is in the environment.
Defining a compiler alias of your own is worth it, so that the sanitizers become the default:
# add to ~/.bashrc or ~/.zshrc
alias cxx='g++ -std=c++20 -g -O1 -Wall -Wextra -fsanitize=address,undefined -fno-omit-frame-pointer'
alias cxxt='g++ -std=c++20 -g -O1 -Wall -Wextra -fsanitize=thread'
cxx hello.cpp -o hello && ./hello
The companion exercises¶
The exercise site has a set of C++ problems (exercises · Advanced C++), graded locally because there is no C++ compiler in the browser:
python practice/judge.py start cpp-raii-fd # copy the template into practice/workspace/
python practice/judge.py test cpp-raii-fd # compile with g++, run the tests under the sanitizers
Found a mistake or a better way to say it? Edit this page on GitHubReport an issue