Introduction to LLM Inference

Let me be honest with you. When I started working on LLM infrastructure, I had eleven years of distributed systems experience. I knew Kafka, Kubernetes, Prometheus. I could debug a partition rebalance in my sleep.

And yet the first time someone asked me what actually happens during inference , I said something like “the model reads the prompt and generates tokens.” Which is technically true the same way “a database reads your query and returns rows” is technically true — accurate, useless, and deeply embarrassing for someone drawing a principal engineer’s salary.

This post is what I wish I’d had on day one. We’re going to walk through the entire inference pipeline — from the moment your request arrives to the moment you see text on screen — with real examples, honest explanations of where the performance goes, and enough detail that you can actually reason about production problems.

No “and then the transformer does its thing.” No skipped steps. Strap in.

Training is where you take a massive dataset, run it through a model millions of times, and slowly adjust billions of numerical weights until the model gets good at predicting the next word. Training is done once (or occasionally). It costs millions of dollars in GPU-hours and requires a team of researchers.

Inference is what happens afterward, every time someone uses the model. It’s the model using those learned weights to respond to new input. No weights change. No learning happens. It’s pure forward-pass computation.

Think of it like this: training is baking the bread. Inference is slicing it and serving it to customers. The bread (weights) is done. The kitchen (inference engine) just has to plate it fast enough that the queue doesn’t back up to the street.

The inference engine is the runtime that takes the frozen model weights and executes them against an input. The same weights can run on Ollama, vLLM, TensorRT-LLM, or TGI — and get meaningfully different performance from each. The weights don’t change. The execution strategy does.

This distinction matters operationally: inference is not a solved problem . Serving a model efficiently at scale is a full engineering discipline.

When you run ollama pull mistral or grab a model from HuggingFace, you aren’t just downloading a “program.” You’re downloading a massive, frozen brain in a box. If you’ve ever wondered why a model that “just chats” takes up 10GB of your SSD, it’s because it is packed with billions of tiny numerical “preferences” the model learned during its training phase.

Think of the GGUF (or Safetensors ) file as a giant Ikea flat-pack box. To build the working model, you need two things: the Instruction Manual and the Hardware .

What’s inside a 7B parameter model file:

GGUF file structure (simplified): ├── Header │ ├── Model architecture (LlamaForCausalLM) │ ├── Vocabulary (32000 tokens + their embeddings) │ ├── Context length (4096, 8192, etc.) │ └── Hyperparameters (n_layers, n_heads, etc.) │ └── Weight tensors: ├── token_embeddings [32000 × 4096] ← the embedding matrix ├── layer.0.attention.q [4096 × 4096] ← Query projection weights ├── layer.0.attention.k [4096 × 4096] ← Key projection weights ├── layer.0.attention.v [4096 × 4096] ← Value projection weights ├── layer.0.attention.out [4096 × 4096] ← Output projection ├── layer.0.ffn.up [4096 × 11008] ← Feed-forward up ├── layer.0.ffn.down [11008 × 4096] ← Feed-forward down ├── ... × 32 layers └── output_norm + lm_head [32000 × 4096] ← Final projection to logits The “Manual” (The Header) This is the first few kilobytes of the file. It tells the inference engine (like Ollama or vLLM) how to put the brain together. It includes:

The rest of that file is just rows and rows of numbers called Weights . Every inference request is essentially looking up values from these matrices and multiplying them together—32 times over.

You might notice some files are 15GB while others are 4GB for the same model. This is Quantization —the art of compression. We turn high-precision 16-bit floats into lower-precision integers (like 4-bit).

Why SREs love INT4: Lower precision = smaller tensors = faster memory transfers. Because decoding is memory-bound, an INT4 model often delivers 20-40% better TPOT (tokens per second) than the “full” version because the memory bus isn’t screaming as loud.

The takeaway: You aren’t executing code; you are loading a massive, math-heavy lookup table. GGUF is your single-file “box,” and quantization is how you fit that box into a smaller truck (your GPU).

Every inference request goes through three broad phases. They are not equally expensive, not equally parallelizable, and not equally friendly to your p99 latency.

┌──────────────────────────────────────────────────────────────────┐ │ TOKENIZATION │ Prefill (Prompt Processing) │ Decode │ │ (CPU) │ (GPU, compute) │ Loop(GPU, Mem) │ │ │ │ │ │ │ │ current → │ │ Text → IDs │ Embed → Position → Attention │ next token │ └──────────────────────────────────────────────────────────────────┘ Fast Scales with prompt length Slow Tokenization : Split the text into token IDs the model understands. CPU-bound. Fast. Prefill : Process the entire prompt through the model. GPU compute-bound. Scales with prompt length. Decode : Generate output tokens one at a time. GPU memory-bound. Runs in a loop until done. Each phase has its own bottleneck. Let’s go through them one by one.

Before a single GPU operation happens, your text has to be converted into a format the model can work with: a sequence of integers called token IDs.

A token is not a character, and it’s not a word. It’s a chunk of text that appears frequently enough in the training corpus to deserve its own ID. There are several ways to build this vocabulary — WordPiece (used by BERT), Unigram (used by SentencePiece), and others — but the dominant approach in modern LLMs is Byte Pair Encoding (BPE): a compression algorithm that iteratively merges the most common pairs of characters or subwords into single tokens until it reaches a target vocabulary size.

The result is a vocabulary of roughly 32,000–128,000 tokens, each with a corresponding integer ID. The model never sees your text — it sees a list of numbers.

After tokenization, this becomes something like:

"The" → 1026 " cat" → 5992 " sat" → 3290 Token IDs: [1026, 5992, 3290]

Note the space before “cat” and “sat” — it’s part of the token. Tokenizers care about whitespace because it affects meaning and frequency.

Yes. The tokenizer is usually written in Rust (HuggingFace’s tokenizers crate) or C++ for exactly this reason. For most requests it’s fast enough to be invisible — microseconds for a short prompt.

Where it bites you: very long documents fed to batch processing jobs. A 100,000-token context requires processing 100,000 token lookups. It’s still fast relative to GPU work, but it’s the one step in the pipeline running on CPU that you can’t just throw more GPU at.

How it’s improved: Parallelizing tokenization across CPU cores for batch workloads. Or — and this is the real fix — not re-tokenizing the same content repeatedly . If you have a shared system prompt you send to every request, tokenizing it once and caching the result is free latency.

Now we have token IDs. The model needs to turn those IDs into something it can reason about. This is prefill — the model processing the entire prompt in one shot.

Prefill has two sub-steps that are easy to conflate: embedding lookup and the actual transformer forward pass . Let’s take them in order.

Every token ID maps to a high-dimensional vector of floating-point numbers called an embedding . These vectors live in the model’s embedding matrix — a giant lookup table with one row per vocabulary token and one column per embedding dimension.

For a model with a 32,000-token vocabulary and 4,096 embedding dimensions, this matrix has shape [32000, 4096] . At 16-bit float precision, that’s about 256MB just for the embedding layer.

Token ID 1026 → embedding row 1026 → [0.12, -0.43, 0.81, ..., 0.07] (4096 values) Token ID 5992 → embedding row 5992 → [-0.34, 0.91, 0.12, ..., -0.22] (4096 values) Token ID 3290 → embedding row 3290 → [0.67, 0.05, -0.88, ..., 0.44] (4096 values) I’m simplifying to 8 dimensions here so this fits on a page. In reality it’s 4,096 or 8,192 dimensions depending on the model.

Simplified (3D instead of 4096D), just to show the shape: "The" → [0.12, -0.43, 0.81] " cat" → [-0.34, 0.91, 0.12] " sat" → [0.67, 0.05, -0.88] Shape: [3 tokens × 3 dims] = a matrix of floats These vectors aren’t random. They’re the result of training — the model has learned that “cat” and “dog” live close together in this space, and “cat” and “quantum mechanics” are far apart. The geometry encodes semantic meaning.

The embedding matrix IS the model weights, specifically. The 10GB (or 40GB, or 70GB) file you download — the GGUF or safetensors file — contains all the weight matrices the model learned during training. The embedding lookup is literally indexing into one of those weight matrices by row number.

When you run inference, you’re not computing anything creative. You’re doing matrix math against frozen numbers that were tuned over millions of training iterations.

Here’s a problem: the embedding lookup is a table lookup. It doesn’t care that “cat” is token 2 and “sat” is token 3. Two requests with the same tokens in different orders would produce identical embeddings.

But order matters enormously. “The cat sat on the dog” and “The dog sat on the cat” have the same tokens and very different meanings.

Positional embeddings solve this by adding a position-aware vector to each token’s embedding. The model learns that “token at position 1” feels different from “token at position 5,” even if the token ID is the same.

Sinusoidal (original Transformers paper): Compute a fixed sine/cosine wave pattern based on position and dimension index. Deterministic, no learned parameters.

RoPE (Rotary Position Embedding): Used by Llama, Qwen, Mistral, and most modern models. Instead of adding a vector, it rotates the query and key vectors by an angle proportional to position. The result: the dot product between two token representations naturally captures their relative distance. Elegant, and generalizes better to longer contexts than the training data.

Continuing our example. After adding positional information:

"The" at position 0: [0.12, -0.43, 0.81] + pos(0) → [0.15, -0.40, 0.84] " cat" at position 1: [-0.34, 0.91, 0.12] + pos(1) → [-0.31, 0.88, 0.09] " sat" at position 2: [0.67, 0.05, -0.88] + pos(2) → [0.65, 0.03, -0.86] The position vectors are small adjustments. Their real value is that when attention is computed later, the model can tell whether two tokens are adjacent or 200 positions apart.

Embedding lookup and positional encoding are fast operations. The real CPU bottleneck in prefill is less about these steps and more about data movement : loading the right weight tensors from CPU RAM to GPU VRAM before the transformer forward pass can begin.

For very large models that don’t fully fit in VRAM, the CPU-GPU transfer becomes the bottleneck — you’re constantly paging weight blocks in. This is why model quantization matters: a 4-bit quantized model uses less VRAM, fits entirely on GPU, and eliminates this transfer overhead. More on that in a moment.

After embedding + positional encoding, we have a matrix of shape [sequence_length × embedding_dim] . This matrix now passes through N transformer layers — 32 layers for Llama-3.2-3B, 80 layers for Llama-3.1-70B.

This is where the model’s “reasoning” happens — and where most of the GPU compute goes during prefill. All tokens are processed in parallel within a layer, making prefill compute-bound. More tokens = more compute = higher TTFT.

We’ll cover the attention mechanism in detail in section 9. First, let’s see what comes out.

After prefill, the model produces its first output token. Then it produces another. Then another. Each token depends on