Math Fundamentals

The math you need when reading about LLMs and inference systems, in a book of its own to look up when you need it: linear algebra and low rank, probability and sampling, information theory, backpropagation and second-order information, floating-point error, rooflines and queueing. Every result is measured on a real Qwen3-0.6B. You do not need to read it first: the other handbooks link here wherever they use this math, so when something stops you, come and read that section.

7 chapters7 exercisesRun on CPU, every result measured on a real Qwen3-0.6B

Start reading→RoadmapExercises

How to use this book

Two ways:

• Look things up: when you hit something from the left column of the table below in another handbook and are not sure about a derivation, open the section on the right, read it, and go back.
• Check where you stand first: take the self-test; its 8 questions cover the whole book, and you can read the chapters behind the ones you cannot answer.

When you read	The math you need	Read this section
Attention	A dot product is a projection; a matrix is a transformation of space	Linear algebra: geometry first
RoPE, rotation-based quantization such as QuaRot	Orthogonal matrices preserve lengths and dot products	Linear algebra: orthogonal matrices and rotations
Tensor parallelism, split-K GEMM	Splitting a matrix multiplication by rows, by columns, or along k	Linear algebra: three views of matrix multiplication
LoRA, MLA	Rank, singular values and low-rank approximation	Linear algebra: rank and low-rank approximation
Decoding and sampling	What temperature, top-k and top-p do to a distribution	Probability and sampling
Speculative decoding	Why rejection sampling leaves the output distribution unchanged, and how to compute the acceptance rate	Probability and sampling: rejection sampling and speculative decoding
The training objective of a language model	Entropy, cross-entropy, perplexity	Information theory: entropy, cross-entropy and KL divergence
How quantization works	How far the distribution moves after quantization, and where the error comes from	Information theory: KL divergence, floating point: quantization noise, calculus: second-order information and GPTQ
Pretraining, training from scratch	The chain rule and backpropagation	Calculus and backpropagation
Normalization and the residual stream, BF16 and FP8	Rounding, accumulation error, overflow	Floating point and numerical computing
Estimating parameters, compute and memory, profiling	Arithmetic intensity and rooflines	Performance math: FLOPs, bytes and arithmetic intensity
Load testing, SLOs and capacity planning	Little's law, queueing theory, tail latency, measurement statistics	Performance math: queueing theory, the statistics of measurement

Chapters

Part	Chapter	What it covers
Math in the model	Linear algebra	What matrices mean geometrically, three ways to split a matrix multiplication, SVD and low rank (why the weights are not low rank but K and V are), orthogonal matrices and rotations
	Probability and sampling	Temperature and top-k / top-p, sampling algorithms, Monte Carlo error, the probability of a sequence, rejection sampling and speculative decoding, importance sampling
	Information theory	Entropy, cross-entropy and KL divergence, where the model is confident, measuring the effect of quantization with KL, forward and reverse KL
	Calculus and backpropagation	The chain rule, reverse-mode automatic differentiation, backpropagation through a linear layer, the second-order information GPTQ uses
Numerics and performance	Floating point and numerical computing	Floating-point formats, rounding and accumulation error, cancellation and overflow, 6 dB of quantization noise per bit
	The math of performance and serving	Arithmetic intensity and rooflines, Amdahl's law, Little's law, queueing theory, tail latency amplification, the statistics of performance measurement
Self-test	Self-test questions	8 questions covering the whole book, each pointing to its section

Code and running it

The examples share an environment with LLM Internals: the mini_llm implemented there plus real Qwen3-0.6B weights, all running on CPU. The code is written to run from the repository's llm/ directory (models/Qwen3-0.6B and the frozen sample text docs/assets/sample-passage.txt are paths relative to llm/); see "Setting up" on the LLM Internals home page for the environment.

To check every example in the book, with output compared line by line against the page:

llm/.venv-llm/bin/python math/tools/check_code.py                         # all 6 chapters
llm/.venv-llm/bin/python math/tools/check_code.py math/docs/probability.md   # check just one chapter

Found a mistake or a better way to say it? Edit this page on GitHub · Report an issue