Local AI Advisor
All lessons
Lesson 7 of 10

Speed: tokens per second

The numbers that decide whether a model feels instant or sluggish.

4 min · you’ll learn

  • What “tok/s” feels like
  • Reading speed vs. writing speed
  • Time to first word
1

Tokens per second: how fast it writes

Speed is measured in tokens per second (tok/s): how many chunks of text the model writes each second once it gets going. Tap the speeds below to feel the difference:

How can I save money on groceries?
Press play to watch the answer appear.

Readable About reading speed. Fine for chat.

0.0s / 7.8s

As a rough guide: under 5 tok/s feels painful, 10–20 is about reading speed, 30+ feels fast. Coding agents want 40+ because they produce lots of text that nobody reads.

2

What decides the speed

Remember lesson 2: for every token the computer re-reads the model. So a simple estimate is memory bandwidth ÷ model size.

A 17 GB model on a Mac with 273 GB/s tops out around 16 tok/s (nearer 12 in practice). The same model on a 1,000 GB/s graphics card: 40–50 tok/s.

3

Reading speed: before it can answer

Before writing anything, the model has to read your whole prompt. This is called prefill or prompt processing, and it’s measured separately.

Think of it like this

Writing speed is how fast someone talks. Reading speed is how long they take to read the letter you handed them before they can reply. Hand them a 50-page letter and you’ll wait.

Pasting a 20,000-token file at 200 tok/s of reading speed means 100 seconds of silence. At 2,000 tok/s it’s 10 seconds. Graphics cards are very good at this; laptops much less so.

4

Time to first token

Time to first token (TTFT) is how long you stare at a blank reply after pressing Enter. It’s the loading time (first message only) plus the reading time.

Enter↑ first tokendone
  • Load model (first request only)
  • Prefill: read your prompt
  • Stream the answer
Illustrative. Everything left of the marker is “time to first token”: you see nothing until it is over.
Want the details?

Runtimes reuse the unchanged start of a conversation (a “prompt cache”), so only new text has to be read again. That makes follow-up messages much faster than the first one.

Remember this

Two speeds matter: how fast it reads your prompt, and how fast it writes the answer. Chat cares about the second; big prompts care about the first.

Quick check

You paste a long document and wait 90 seconds before the first word appears. Which speed is the bottleneck?

Up next · lesson 8

Chat vs. coding agents

Why a model that’s great for chat can be painfully slow as a coding agent.

Context: the model’s short-term memoryAll lessons