Context: the model’s short-term memory
How much text a model can keep in mind at once, and what that costs.
4 min · you’ll learn
- What tokens are
- How big a context you need
- Why long chats use more memory
Models read and write in tokens
Models don’t see letters or whole words. They see tokens: chunks of text about ¾ of an English word long (or about 4 characters of code).
What you type
Local models run on your own computer, even offline!
What the model sees: 12 tokens
The context window: how much it can keep in mind
The context window is how many tokens the model can consider at once: your messages, its replies, any files you paste, all of it.
- Instructions · 2K
- Your messages · 3K
- Its replies · 5K
- A pasted file · 12K
- Still free
- 8K tokens
- ≈ an article
- casual chat
- 32K tokens
- ≈ a few dozen files
- coding help
- 128K tokens
- ≈ a short book
- long documents
Think of it like this
Context is the model’s desk. A bigger desk holds more papers at once, but it takes up more room in the office (memory).
Context costs memory: the KV cache
To avoid re-reading the whole conversation for every word, the model keeps notes on each token it has seen. These notes are called the KV cache, and they live in memory right next to the model.
The longer the context, the bigger the cache. At long contexts it can grow to several gigabytes, on top of the model itself.
Live · uses the real engine
KV cache vs. context length
Extra memory needed on top of the weights for one conversation.
Hybrid linear attention
Only ~25% of layers keep a per-token KV cache; the others keep a small fixed-size state, whatever the context length.
At 128K: 8.0 GB for Qwen3.6 27B vs. 32 GB for Qwen3 32B (4.0× less).
Want the details?
Newer “hybrid” models use sliding-window or linear attention in some layers, so their cache grows much more slowly. Runtimes like llama.cpp can also store the cache at 8-bit (“Q8 KV”), roughly halving it with minor quality impact.
Don’t max it out
A model might advertise a 256K context. That’s the most it can handle, not what you should run. Every token of context costs memory and slows down reading your prompt. Pick what your task needs: 8–16K for chat, 32–64K for coding agents.
Remember this
Quick check
Your chat app gets slow and runs out of memory during a very long conversation. Why?
Up next · lesson 7
Speed: tokens per second
The numbers that decide whether a model feels instant or sluggish.