Memory: where the model lives
A model has to sit in memory while it runs, and what kind of memory decides how fast it is.
4 min · you’ll learn
- RAM vs. VRAM in plain words
- Why Macs can run big models
- Why bandwidth beats raw power
A model has to sit in memory while it runs
A model is a big file: often 5 to 40 GB. To use it, your computer loads the whole thing into memory, and to write every single word it has to read through (almost) all of it again.
Think of it like this
It’s like a cook who re-reads the entire recipe book before adding each ingredient. The book has to be on the counter (in memory), and how fast the cook can flip pages decides how fast dinner arrives.
RAM: your computer’s main memory
RAM is the working memory every computer has, shared by the operating system and all your apps. It’s plentiful and cheap, but comparatively slow to read: a typical desktop manages 60–100 GB per second.
That “GB per second” number is called memory bandwidth, and for AI it matters more than how fast your processor is.
VRAM: the graphics card’s own memory
A graphics card (GPU) has its own memory soldered onto it, called VRAM. It is small and expensive, but very fast: often 10× faster than RAM.
How fast each kind of memory can be read
- Desktop RAM (DDR5)≈80 GB/s
- MacBook Pro M4 Pro≈273 GB/s
- Mac Studio M4 Max≈546 GB/s
- RTX 4090 graphics card≈1,008 GB/s
If the model fits entirely in VRAM, it runs fast. If it’s too big, the extra part spills into slower RAM and everything slows down sharply. An RTX 4090 has 24 GB of VRAM: a 20 GB model flies, a 30 GB model crawls, even if the PC has 64 GB of RAM.
Unified memory: one big shared pool
Apple Silicon Macs (and a few PCs like AMD Ryzen AI Max and NVIDIA DGX Spark) use unified memory: one pool shared by the processor and the graphics chip. There’s no small VRAM limit to overflow, so a 64 GB Mac can run models that would never fit on a 24 GB graphics card.
PC with a graphics card
Fast. The model should live here.
OS, apps, and any model layers that didn’t fit in VRAM.
Unified memory (one pool)
CPU and GPU share it. No copying, no spill cliff, but the OS and your apps compete for the same space.
Apple Silicon, AMD Ryzen AI Max, NVIDIA DGX Spark
The fine print on Macs
The graphics chip isn’t allowed to use all of unified memory. By default macOS lets it use about two-thirds on smaller Macs and about three-quarters on bigger ones. So a 48 GB MacBook Pro gives the GPU roughly 36 GB.
Advanced users can raise the limit with sudo sysctl iogpu.wired_limit_mb=…, at the cost of leaving less memory for macOS and your apps.
Remember this
Quick check
A gaming PC has a 12 GB graphics card and 64 GB of RAM. You load a 20 GB model. What happens?
Up next · lesson 3
How big is a model?
What “7B” or “35B-A3B” means, and how size turns into gigabytes.