LLM Internals
Putting it together
Learning resources¶
This handbook is a "map". To go deeper in any direction, you need to read the original papers and source code. Below, the most worthwhile material is listed by chapter, each item with why it is worth reading and what to watch for; at the end come the next steps toward inference optimization after finishing this handbook.
How to read papers¶
- Figures and tables first, then formulas, then the text. The core of an architecture paper is often one structure diagram and one ablation table.
- Read with questions: what bottleneck does this design address? What price does it pay? (Compare with the cause-and-effect picture in the complete journey of a token.)
- Reproduce what you can: every concept in this handbook has a runnable piece of code, and when reading a paper you can also write a minimal implementation and check it against the paper's numbers.
By chapter¶
Basics and the Transformer¶
| Material | Why read it |
|---|---|
| Attention Is All You Need (2017) | The original Transformer paper. Note the encoder-decoder structure, Post-Norm and sinusoidal positional encoding, all since changed by later models, and think about why |
| The Illustrated Transformer | Illustrated, good for building intuition on a first pass |
| Neural Machine Translation of Rare Words with Subword Units (BPE) | Where BPE started in NLP |
| karpathy/minbpe, karpathy/nanoGPT, karpathy/nanochat | The most readable minimal tokenizer, GPT training code, and a complete small pipeline from pretraining to chat; the accompanying videos are well worth watching too |
| RoFormer (RoPE) | The derivation of RoPE; section 3 is enough |
| Root Mean Square Layer Normalization | RMSNorm |
| GLU Variants Improve Transformer | Where SwiGLU comes from, only 3 pages |
| A Mathematical Framework for Transformer Circuits | Understanding the Transformer from the "residual stream" point of view, echoing this handbook's residual stream section |
| transformers' modeling_llama.py | Compare it line by line with this handbook's mini_llm.py, a shortcut to understanding every LLaMA-style model implementation |
How architectures evolved¶
| Material | Why read it |
|---|---|
| Fast Transformer Decoding: One Write-Head is All You Need (MQA) | One of the first papers to design model structure starting from "decode is memory bound" |
| GQA | GQA, and how to convert MHA checkpoints to GQA |
| DeepSeek-V2 | Where MLA comes from; section 2.1 details the low-rank compression, weight absorption and decoupled RoPE |
| DeepSeek-V3 | Fine-grained MoE, auxiliary-loss-free load balancing, MTP, FP8 training, and the inference deployment plan (section 3.4) |
| Switch Transformers, Mixtral of Experts | MoE routing, capacity and load balancing |
| LLaMA, Llama 2, The Llama 3 Herd of Models | Where the standard template came from; the Llama 3 report describes data, training infrastructure and inference (FP8, pipeline parallelism) in detail |
| Qwen2.5, Qwen3 technical reports | The "family tree" of the model this handbook uses |
| Gemma 2, Gemma 3 | Local/global attention mixes, soft-capping, QK-Norm |
| gpt-oss model card | Attention sinks, alternating sliding windows, MXFP4 |
| YaRN | The most common RoPE scaling method for long-context extension |
Training and alignment¶
| Material | Why read it |
|---|---|
| Scaling Laws for Neural Language Models, Chinchilla | \(6ND\), the compute-optimal ratio of parameters to data |
| InstructGPT | The classic SFT + RLHF pipeline |
| DPO | Follow the derivation of the loss and you understand preference optimization |
| DeepSeekMath (GRPO), DeepSeek-R1 | How reasoning models are trained with reinforcement learning; the massive sampling in RL training is a new battlefield for inference engines |
| LoRA | Low-rank fine-tuning, and the basis of multi-LoRA serving |
How inference works, and inference serving¶
| Material | Why read it |
|---|---|
| Transformer Inference Arithmetic | The classic blog post on inference estimation, complementing this handbook's estimation chapter |
| How to Scale Your Model | Google DeepMind's "roofline + parallelism" tutorial; the inference chapter is especially worth reading, from a TPU angle but with general principles |
| FlashAttention, FlashAttention-2, FlashAttention-3 | IO-aware attention; read with FlashAttention and inference operators in the CUDA book |
| Efficient Memory Management for LLM Serving with PagedAttention (vLLM) | The design of PagedAttention and vLLM |
| SGLang | RadixAttention prefix caching, compressed state machines for structured output |
| Sarathi-Serve | Chunked prefill and "stall-free" scheduling |
| DistServe, Mooncake | PD disaggregation; Mooncake is a production architecture centered on the KV cache |
| Fast Inference from Transformers via Speculative Decoding, Accelerating LLM Decoding with Speculative Sampling | Speculative decoding and the proof that speculative sampling is correct |
| Medusa, EAGLE | Speculative decoding without a separate draft model |
| LLM.int8(), SmoothQuant, GPTQ, AWQ | The outlier problem and the mainstream quantization methods |
| Efficient Streaming Language Models with Attention Sinks, Massive Activations in LLMs | The attention sinks and massive activations this handbook observed in Qwen |
| Defeating Nondeterminism in LLM Inference | Why results change even at temperature 0, and how to build batch-invariant kernels |
After this handbook: toward inference optimization¶
This handbook covers "what the model computes". The next two parts:
1. How the hardware executes it: CUDA. See Advanced CUDA. The chapters most closely tied to this handbook:
- The road to GEMM optimization: every linear layer;
- Softmax and normalization: RMSNorm, fused kernels;
- FlashAttention and inference operators: prefill and paged decode attention;
- Quantization and GEMV: decode with INT4 weights;
- Streams, concurrency and CUDA Graphs, multi-GPU and NCCL.
2. How the system schedules it: the fourth book of this series, Inference Systems, is devoted to this part: write an inference engine from scratch, then read the vLLM and SGLang source against it, followed by distributed inference, performance engineering and interview preparation. To go straight to the source code, the suggested order:
- nano-vllm: implements the core of vLLM (paged KV, continuous batching, prefix caching, CUDA Graphs, tensor parallelism) in a little over a thousand lines; readable in a day or two, the best starting point for understanding a complete engine;
- vLLM: start from a request's lifecycle (API server → engine → scheduler → model executor → sampler), then look at the model implementations you know under
model_executor/models/, compared againstmini_llm.py; - SGLang: focus on the scheduler (prefix caching, overlap scheduling) and the DeepSeek-related optimizations (MLA, DP attention, EP).
When reading source code, use this handbook's "stop by stop" table (see the complete journey of a token) as an index: for every module you meet, ask which stop of the table it corresponds to and which bottleneck it addresses.
3. Build projects. A few directions, from easy to hard:
- Add a paged KV cache and continuous batching to
mini_llm.py, writing a "mini inference engine" of a few hundred lines, and measure how throughput changes with batch size; - Write fused kernels for RMSNorm, SwiGLU and RoPE in Triton on a GPU, replace the corresponding parts of
mini_llm.py, and compare speed and accuracy; - Implement (or reproduce) a small feature in vLLM or SGLang: a new sampling parameter, support for a new model structure, or a kernel optimization, and submit a PR.
Summary¶
- Read papers figures and tables first, with the questions "which bottleneck does it address, and at what price".
- Every chapter has corresponding original papers; the inference papers are the focus for going deeper.
- Next: the CUDA book covers "how the hardware executes it", and the inference systems book (plus the source code of nano-vllm → vLLM → SGLang) covers "how the system schedules it".
Found a mistake or a better way to say it? Edit this page on GitHubReport an issue