Methodology
How the advisor turns your hardware, tool and workload into a comfort rating — and where every number comes from.
The catalog stores vendor-documented practical context. Live Hugging Face imports show native context plus any RoPE-extended window, and rate against native context; quality past native may degrade, so long-context ratings there are optimistic.
How the engine works
Every evaluation of a (hardware, model, quantization, runtime, tool, workload) combination runs the same 17-step pipeline. Nothing is pre-labelled: recommendation status is always derived, never stored.
- 1Validate hardware supportIs the operating system supported by the machine, and is there a usable GPU?
- 2Validate runtime supportDoes the runtime ship for this OS and have a backend (Metal, CUDA, ROCm, Vulkan, SYCL or CPU) for this GPU?
- 3Validate tool → API → runtimeCan the AI tool reach the runtime over an API it speaks — directly, or through a bridge?
- 4Estimate usable RAM / VRAMUnified memory is capped by the GPU wired-memory limit; discrete GPUs use VRAM first, system RAM second.
- 5Reserve OS / dev-env memoryMemory for the OS, your IDE, browsers, containers and the tool itself is set aside before the model gets any.
- 6WeightsModel size at the chosen quantization and format, using published file sizes where available.
- 7Runtime overheadCompute buffers and fixed runtime overhead, growing with batch size, context and parallel streams.
- 8KV / context memoryPer-token KV cache × context × streams, honouring sliding-window and hybrid attention and KV quantization.
- 9Determine memory headroomWhat is left after everything above — too little headroom means swapping, crashes or a sluggish machine.
- 10Estimate prompt processingPrefill speed (compute bound) and how long cold prompts and follow-up steps take to ingest.
- 11Estimate generationDecode speed (bandwidth bound) at short, typical and full context.
- 12Apply workload requirementsCompare speeds, latencies, context and headroom against the thresholds of your use case.
- 13Apply concurrencyParallel requests and agents slow each stream depending on how well the runtime batches.
- 14Model suitabilityEditorial capability tiers, tool-calling reliability and vision support versus what the use case needs.
- 15ConfidenceHow much of the estimate rests on measurements versus specifications and assumptions.
- 16Comfort classificationWeighted dimensions are combined; weak critical dimensions cap the overall rating.
- 17Human-readable explanationPositives, warnings, bottlenecks and why the rating is not one level higher.
Comfort levels
The advisor answers “will it be comfortable?”, not just “does it load?”. The same model on the same machine can be Excellent for chat and Borderline for agentic coding.
- ExcellentSubstantial headroom for this workload. Expect a highly responsive experience.
- ComfortableA good practical setup for this workload: reasonable latency, memory headroom and context.
- AcceptableUsable, but you will notice compromises.
- BorderlineTechnically usable, but likely frustrating for this workload.
- Technically RunsLoads and executes, but the experience is inappropriate for this workload.
- Does Not FitNot enough usable memory for this model at these settings.
- UnsupportedThis hardware / runtime / tool combination does not work.
Weights differ per use case
Each use case weights the nine dimensions differently, and sets its own generation-speed ladder. Speeds are per-stream decode tokens/second needed for Excellent / Comfortable / Acceptable / Borderline.
| Use case | Memory | Generation | Prompt processing | Context | Runtime | Tool compat. | Suitability | Concurrency | Sustained use | Generation tok/sExc. / Comf. / Acc. / Bord. |
|---|---|---|---|---|---|---|---|---|---|---|
| Casual chatChat & writing | high (caps the rating) | critical (caps the rating) | medium | medium | medium | medium | medium | — | medium | 35 / 16 / 9 / 5 |
| General assistantChat & writing | high (caps the rating) | critical (caps the rating) | medium | medium | medium | medium | high | — | medium | 38 / 18 / 10 / 5 |
| Coding questionsCoding | high (caps the rating) | critical (caps the rating) | high | medium | medium | medium | critical (caps the rating) | — | medium | 45 / 22 / 13 / 7 |
| Coding with repository contextCoding | very high (caps the rating) | critical (caps the rating) | critical (caps the rating) | high | medium | high (caps the rating) | critical (caps the rating) | — | medium | 50 / 28 / 17 / 9 |
| Agentic codingCoding | critical (caps the rating) | critical (caps the rating) | critical (caps the rating) | very high (caps the rating) | high | critical (caps the rating) | critical (caps the rating) | medium | high | 60 / 35 / 22 / 12 |
| ReasoningDocuments & research | high (caps the rating) | critical (caps the rating) | medium | high | medium | medium | critical (caps the rating) | — | medium | 50 / 28 / 16 / 8 |
| ResearchDocuments & research | high (caps the rating) | high | very high (caps the rating) | very high (caps the rating) | medium | medium | very high | — | medium | 40 / 20 / 11 / 6 |
| Document analysisDocuments & research | very high (caps the rating) | medium | critical (caps the rating) | critical (caps the rating) | medium | medium | high | — | medium | 30 / 14 / 8 / 4 |
| Long-document Q&ADocuments & research | critical (caps the rating) | medium | critical (caps the rating) | critical (caps the rating) | medium | medium | high | — | medium | 30 / 14 / 8 / 4 |
| WritingChat & writing | high (caps the rating) | very high (caps the rating) | medium | medium | medium | medium | very high | — | medium | 35 / 16 / 9 / 5 |
| Vision / image understandingMultimodal | high (caps the rating) | very high (caps the rating) | high | medium | medium | medium | critical (caps the rating) | — | medium | 35 / 16 / 9 / 5 |
| Data analysisDocuments & research | high | very high (caps the rating) | very high (caps the rating) | high | medium | medium | critical (caps the rating) | — | medium | 45 / 24 / 14 / 7 |
| RAG (chat with your files)Documents & research | high (caps the rating) | high (caps the rating) | critical (caps the rating) | high | medium | medium | high | — | medium | 38 / 18 / 10 / 5 |
| Local API serverAutomation & serving | critical (caps the rating) | very high | high | medium | very high (caps the rating) | medium | medium | critical (caps the rating) | high | 40 / 22 / 12 / 6 |
| Multiple agentsAutomation & serving | critical (caps the rating) | critical (caps the rating) | critical (caps the rating) | very high | very high | very high (caps the rating) | critical (caps the rating) | critical (caps the rating) | high | 45 / 28 / 18 / 10 |
| Background automationAutomation & serving | critical (caps the rating) | medium | medium | medium | medium | medium | high | — | very high (caps the rating) | 25 / 12 / 6 / 3 |
* Dimensions marked with an asterisk cap the overall rating when they are weak — a model can’t be Comfortable for agentic coding if prompt processing is poor, however fast it generates.
Performance model
Generation (decode) is bandwidth bound
Every new token reads the model’s active weights plus the KV cache from memory. Tokens/second ≈ effective bandwidth ÷ bytes read per token. Mixture-of-experts models read only their active experts, which is why a 30B-A3B model generates far faster than a dense 30B. Speed drops as the context fills because the KV cache must be read too.
Prompt processing (prefill) is compute bound
Ingesting a prompt costs roughly 2 × active parameters FLOPs per token, plus attention cost that grows with context. GPU TFLOPS and the runtime’s kernel efficiency set the pace. This dominates agentic tools that resend large system prompts, files and tool output every step; prompt-cache reuse in the runtime reduces it.
Calibration and the basis of every number
- Measured
A verified benchmark exists for this exact chip, model and quantization. Shown as “≈N tok/s”.
- Calibrated
Estimated from specs, then corrected using verified benchmarks of other models on the same chip. Shown as a ±15% range.
- Estimated
No benchmark applies — derived from bandwidth, compute and runtime efficiency. Shown as a ±25% range.
Ranges are deliberate: we never show a falsely precise number for an estimate, and never invent a “measured” value.
Memory model
- Peak memory = weights + KV cache + runtime overhead + vision encoder (if images are used), plus memory reserved for the OS, your development environment and the AI tool.
- Weights use published GGUF file sizes when available; otherwise parameters × effective bits per weight (including scales) for GGUF, MLX or safetensors.
- KV cache = 2 × layers × KV heads × head dim × bytes, per token of context and per parallel stream. Sliding-window and hybrid linear-attention models only keep a full cache on some layers; KV quantization (Q8 / Q4) shrinks it where the runtime supports it.
- Unified memory is one pool with a GPU wired-memory cap (≈⅔–¾ of RAM on macOS by default, adjustable). Discrete GPUs have VRAM and system RAM as separate pools; weights that spill out of VRAM run at system-RAM speed and slow decode sharply.
Confidence
Each result carries a High / Medium / Low confidence, starting from its performance basis and adjusted for known sources of uncertainty.
- Starting point: measured (highest) → calibrated → estimated (lowest).
- Lower when the runtime backend is experimental.
- Lower for long contexts (over 32K), where performance is extrapolated.
- Lower with partial GPU offload, which is hard to predict precisely.
- Lower for unmeasured MoE and hybrid/sliding-attention models, whose speed varies more between runtimes.
- Lower when hardware or model specs come from low-confidence sources, or prefill gains rely on vendor claims (e.g. new GPU matrix accelerators).
Compatibility and memory-fit results are deterministic rules and are always high confidence.
Compatibility rules
Blockers make a combination Unsupported; warnings lower the rating or add caveats.
| Layer | Rule | Severity |
|---|---|---|
| Hardware ↔ runtime | The runtime must ship for the selected operating system. | blocker |
| Hardware ↔ runtime | MLX-based runtimes require Apple Silicon. | blocker |
| Hardware ↔ runtime | If no GPU backend of the runtime matches the hardware, inference falls back to the CPU (much slower). | warning |
| Runtime ↔ model | The runtime must load a format the model is published in (GGUF, MLX, safetensors). | blocker |
| Runtime ↔ model | The quantization must exist for that format (e.g. vLLM does not load GGUF Q5_K_M). | blocker |
| Tool ↔ runtime | The tool must speak an API the runtime exposes (OpenAI-compatible, Anthropic-compatible or Ollama), possibly through a bridge. | blocker |
| Tool ↔ runtime | Agentic tools need a minimum context window for their system prompt and tool definitions. | warning |
| Model ↔ use case | Image understanding requires a vision-capable model. | blocker |
| Model ↔ use case | Agentic tools need reliable structured tool calling from the model. | warning |
Benchmark data
Verified public llama.cpp results transcribed from the sources linked below. pp = prompt processing (prefill), tg = text generation. They anchor the performance model as calibration points.
| Chip | Model | Quant | Backend | pp / tg tokens | Prefill tok/s | Generation tok/s | Source | Date |
|---|---|---|---|---|---|---|---|---|
| Apple M1 (8-core GPU) | LLaMA 2 7B (benchmark reference) | Q4_0 | Metal (Apple GPU) | 512 / 128 | 118.0 | 14.2 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M1 (8-core GPU) | LLaMA 2 7B (benchmark reference) | Q8_0 | Metal (Apple GPU) | 512 / 128 | 117.3 | 7.9 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M1 Max (32-core GPU) | LLaMA 2 7B (benchmark reference) | Q4_0 | Metal (Apple GPU) | 512 / 128 | 530.1 | 61.2 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M1 Max (32-core GPU) | LLaMA 2 7B (benchmark reference) | Q8_0 | Metal (Apple GPU) | 512 / 128 | 537.4 | 40.2 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M1 Pro (16-core GPU) | LLaMA 2 7B (benchmark reference) | Q4_0 | Metal (Apple GPU) | 512 / 128 | 266.3 | 36.4 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M1 Pro (16-core GPU) | LLaMA 2 7B (benchmark reference) | Q8_0 | Metal (Apple GPU) | 512 / 128 | 270.4 | 22.3 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M1 Ultra (64-core GPU) | LLaMA 2 7B (benchmark reference) | Q4_0 | Metal (Apple GPU) | 512 / 128 | 1030.0 | 83.7 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M1 Ultra (64-core GPU) | LLaMA 2 7B (benchmark reference) | Q8_0 | Metal (Apple GPU) | 512 / 128 | 1043.0 | 59.9 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M2 (10-core GPU) | LLaMA 2 7B (benchmark reference) | Q4_0 | Metal (Apple GPU) | 512 / 128 | 179.6 | 21.9 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M2 (10-core GPU) | LLaMA 2 7B (benchmark reference) | Q8_0 | Metal (Apple GPU) | 512 / 128 | 181.4 | 12.2 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M2 Max (38-core GPU) | LLaMA 2 7B (benchmark reference) | Q4_0 | Metal (Apple GPU) | 512 / 128 | 671.3 | 66.0 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M2 Max (38-core GPU) | LLaMA 2 7B (benchmark reference) | Q8_0 | Metal (Apple GPU) | 512 / 128 | 677.9 | 41.8 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M2 Pro (19-core GPU) | LLaMA 2 7B (benchmark reference) | Q4_0 | Metal (Apple GPU) | 512 / 128 | 341.2 | 38.9 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M2 Pro (19-core GPU) | LLaMA 2 7B (benchmark reference) | Q8_0 | Metal (Apple GPU) | 512 / 128 | 344.5 | 23.0 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M2 Ultra (76-core GPU) | gpt-oss-20b | MXFP4 | Metal (Apple GPU) | 2048 / 32 | 2713.4 | 130.0 | llama.cpp repo benches — Mac M2 Ultra (build 7948)(opens in a new tab) | 2026 |
| Apple M2 Ultra (76-core GPU) | LLaMA 2 7B (benchmark reference) | Q4_0 | Metal (Apple GPU) | 512 / 128 | 1238.5 | 94.3 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M2 Ultra (76-core GPU) | gpt-oss-120b | MXFP4 | Metal (Apple GPU) | 2048 / 32 | 1648.7 | 85.6 | llama.cpp repo benches — Mac M2 Ultra (build 7948)(opens in a new tab) | 2026 |
| Apple M2 Ultra (76-core GPU) | Qwen2.5-Coder 7B Instruct | Q8_0 | Metal (Apple GPU) | 2048 / 32 | 1565.9 | 79.7 | llama.cpp repo benches — Mac M2 Ultra (build 7948)(opens in a new tab) | 2026 |
| Apple M2 Ultra (76-core GPU) | Qwen3-Coder 30B-A3B Instruct | Q8_0 | Metal (Apple GPU) | 2048 / 32 | 2453.1 | 79.0 | llama.cpp repo benches — Mac M2 Ultra (build 7948)(opens in a new tab) | 2026 |
| Apple M2 Ultra (76-core GPU) | LLaMA 2 7B (benchmark reference) | Q8_0 | Metal (Apple GPU) | 512 / 128 | 1248.6 | 66.6 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M2 Ultra (76-core GPU) | GLM-4.7-Flash (30B-A3B) | Q8_0 | Metal (Apple GPU) | 2048 / 32 | 1629.3 | 59.6 | llama.cpp repo benches — Mac M2 Ultra (build 7948)(opens in a new tab) | 2026 |
| Apple M3 (10-core GPU) | LLaMA 2 7B (benchmark reference) | Q4_0 | Metal (Apple GPU) | 512 / 128 | 186.8 | 21.3 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M3 (10-core GPU) | LLaMA 2 7B (benchmark reference) | Q8_0 | Metal (Apple GPU) | 512 / 128 | 187.5 | 12.3 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M3 Max (30-core GPU) | LLaMA 2 7B (benchmark reference) | Q4_0 | Metal (Apple GPU) | 512 / 128 | 567.6 | 56.6 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M3 Max (30-core GPU) | LLaMA 2 7B (benchmark reference) | Q8_0 | Metal (Apple GPU) | 512 / 128 | 566.4 | 34.3 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M3 Max (40-core GPU) | LLaMA 2 7B (benchmark reference) | Q4_0 | Metal (Apple GPU) | 512 / 128 | 759.7 | 66.3 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M3 Max (40-core GPU) | LLaMA 2 7B (benchmark reference) | Q8_0 | Metal (Apple GPU) | 512 / 128 | 757.6 | 42.8 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M3 Pro (18-core GPU) | LLaMA 2 7B (benchmark reference) | Q4_0 | Metal (Apple GPU) | 512 / 128 | 341.7 | 30.7 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M3 Pro (18-core GPU) | LLaMA 2 7B (benchmark reference) | Q8_0 | Metal (Apple GPU) | 512 / 128 | 344.7 | 17.5 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M3 Ultra (80-core GPU) | gpt-oss-20b | MXFP4 | Metal (Apple GPU) | 2048 / 128 | 2816.5 | 115.5 | llama.cpp — gpt-oss guide & performance (Aug 2025)(opens in a new tab) | 2025-08 |
| Apple M3 Ultra (80-core GPU) | LLaMA 2 7B (benchmark reference) | Q4_0 | Metal (Apple GPU) | 512 / 128 | 1471.2 | 92.1 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M3 Ultra (80-core GPU) | LLaMA 2 7B (benchmark reference) | Q8_0 | Metal (Apple GPU) | 512 / 128 | 1487.5 | 63.9 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M4 (10-core GPU) | LLaMA 2 7B (benchmark reference) | Q4_0 | Metal (Apple GPU) | 512 / 128 | 221.3 | 24.1 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M4 (10-core GPU) | LLaMA 2 7B (benchmark reference) | Q8_0 | Metal (Apple GPU) | 512 / 128 | 223.6 | 13.5 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M4 Max (32-core GPU) | gpt-oss-20b | MXFP4 | Metal (Apple GPU) | 2048 / 128 | 1277.4 | 92.4 | llama.cpp — gpt-oss guide & performance (Aug 2025)(opens in a new tab) | 2025-08 |
| Apple M4 Max (32-core GPU) | LLaMA 2 7B (benchmark reference) | Q4_0 | Metal (Apple GPU) | 512 / 128 | 713.9 | 70.0 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M4 Max (32-core GPU) | LLaMA 2 7B (benchmark reference) | Q8_0 | Metal (Apple GPU) | 512 / 128 | 718.6 | 43.9 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M4 Max (40-core GPU) | LLaMA 2 7B (benchmark reference) | Q4_0 | Metal (Apple GPU) | 512 / 128 | 885.7 | 83.1 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M4 Max (40-core GPU) | LLaMA 2 7B (benchmark reference) | Q8_0 | Metal (Apple GPU) | 512 / 128 | 891.9 | 54.0 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M4 Pro (16-core GPU) | LLaMA 2 7B (benchmark reference) | Q4_0 | Metal (Apple GPU) | 512 / 128 | 364.1 | 49.6 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M4 Pro (16-core GPU) | LLaMA 2 7B (benchmark reference) | Q8_0 | Metal (Apple GPU) | 512 / 128 | 367.1 | 30.5 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M4 Pro (20-core GPU) | LLaMA 2 7B (benchmark reference) | Q4_0 | Metal (Apple GPU) | 512 / 128 | 439.8 | 50.7 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| Apple M4 Pro (20-core GPU) | LLaMA 2 7B (benchmark reference) | Q8_0 | Metal (Apple GPU) | 512 / 128 | 449.6 | 30.7 | llama.cpp — Performance of llama.cpp on Apple Silicon M-series(opens in a new tab) | 2023-11 → 2025 (community table) |
| GB10 Blackwell GPU | gpt-oss-20b | MXFP4 | CUDA (NVIDIA) | 2048 / 32 | 4505.8 | 83.4 | llama.cpp repo benches — DGX Spark (build 7941)(opens in a new tab) | 2026 |
| GB10 Blackwell GPU | Qwen3-Coder 30B-A3B Instruct | Q8_0 | CUDA (NVIDIA) | 2048 / 32 | 2987.0 | 61.1 | llama.cpp repo benches — DGX Spark (build 7941)(opens in a new tab) | 2026 |
| GB10 Blackwell GPU | gpt-oss-120b | MXFP4 | CUDA (NVIDIA) | 2048 / 32 | 2443.9 | 58.7 | llama.cpp repo benches — DGX Spark (build 7941)(opens in a new tab) | 2026 |
| GB10 Blackwell GPU | LLaMA 2 7B (benchmark reference) | Q4_0 | CUDA (NVIDIA) | 512 / 128 | 3062.3 | 57.2 | llama.cpp — CUDA performance scoreboard(opens in a new tab) | 2025-08 → 2026 |
| GB10 Blackwell GPU | GLM-4.7-Flash (30B-A3B) | Q8_0 | CUDA (NVIDIA) | 2048 / 32 | 2364.2 | 48.7 | llama.cpp repo benches — DGX Spark (build 7941)(opens in a new tab) | 2026 |
| GB10 Blackwell GPU | Qwen2.5-Coder 7B Instruct | Q8_0 | CUDA (NVIDIA) | 2048 / 32 | 2250.3 | 29.4 | llama.cpp repo benches — DGX Spark (build 7941)(opens in a new tab) | 2026 |
| GeForce RTX 3090 | LLaMA 2 7B (benchmark reference) | Q4_0 | Vulkan | 512 / 128 | 4666.1 | 164.1 | llama.cpp — Vulkan performance scoreboard(opens in a new tab) | 2025 → 2026 |
| GeForce RTX 3090 | gpt-oss-20b | MXFP4 | CUDA (NVIDIA) | 2048 / 128 | 5170.6 | 161.8 | llama.cpp — gpt-oss guide & performance (Aug 2025)(opens in a new tab) | 2025-08 |
| GeForce RTX 3090 | LLaMA 2 7B (benchmark reference) | Q4_0 | CUDA (NVIDIA) | 512 / 128 | 5174.7 | 158.2 | llama.cpp — CUDA performance scoreboard(opens in a new tab) | 2025-08 → 2026 |
| GeForce RTX 4080 SUPER | gpt-oss-20b | MXFP4 | CUDA (NVIDIA) | 2048 / 128 | 8170.9 | 186.5 | llama.cpp — gpt-oss guide & performance (Aug 2025)(opens in a new tab) | 2025-08 |
| GeForce RTX 4090 | gpt-oss-20b | MXFP4 | CUDA (NVIDIA) | 2048 / 128 | 8022.3 | 225.2 | llama.cpp — gpt-oss guide & performance (Aug 2025)(opens in a new tab) | 2025-08 |
| GeForce RTX 4090 | LLaMA 2 7B (benchmark reference) | Q4_0 | Vulkan | 512 / 128 | 9452.0 | 188.0 | llama.cpp — Vulkan performance scoreboard(opens in a new tab) | 2025 → 2026 |
| GeForce RTX 4090 | LLaMA 2 7B (benchmark reference) | Q4_0 | CUDA (NVIDIA) | 512 / 128 | 11992.7 | 186.2 | llama.cpp — CUDA performance scoreboard(opens in a new tab) | 2025-08 → 2026 |
| GeForce RTX 5060 Ti 16GB | gpt-oss-20b | MXFP4 | CUDA (NVIDIA) | 2048 / 128 | — | 111.5 | llama.cpp — gpt-oss guide & performance (Aug 2025)(opens in a new tab) | 2025-08 |
| GeForce RTX 5060 Ti 16GB | LLaMA 2 7B (benchmark reference) | Q4_0 | CUDA (NVIDIA) | 512 / 128 | 3737.3 | 90.9 | llama.cpp — CUDA performance scoreboard(opens in a new tab) | 2025-08 → 2026 |
| GeForce RTX 5070 Ti | gpt-oss-20b | MXFP4 | CUDA (NVIDIA) | 2048 / 128 | 6339.8 | 189.4 | llama.cpp — gpt-oss guide & performance (Aug 2025)(opens in a new tab) | 2025-08 |
| GeForce RTX 5070 Ti | LLaMA 2 7B (benchmark reference) | Q4_0 | CUDA (NVIDIA) | 512 / 128 | 6952.4 | 176.8 | llama.cpp — CUDA performance scoreboard(opens in a new tab) | 2025-08 → 2026 |
| GeForce RTX 5080 | gpt-oss-20b | MXFP4 | CUDA (NVIDIA) | 2048 / 128 | 7476.6 | 204.8 | llama.cpp — gpt-oss guide & performance (Aug 2025)(opens in a new tab) | 2025-08 |
| GeForce RTX 5080 | LLaMA 2 7B (benchmark reference) | Q4_0 | CUDA (NVIDIA) | 512 / 128 | 8297.4 | 182.0 | llama.cpp — CUDA performance scoreboard(opens in a new tab) | 2025-08 → 2026 |
| GeForce RTX 5090 | LLaMA 2 7B (benchmark reference) | Q4_0 | CUDA (NVIDIA) | 512 / 128 | 14073.4 | 290.0 | llama.cpp — CUDA performance scoreboard(opens in a new tab) | 2025-08 → 2026 |
| GeForce RTX 5090 | gpt-oss-20b | MXFP4 | CUDA (NVIDIA) | 2048 / 128 | 9848.4 | 282.5 | llama.cpp — gpt-oss guide & performance (Aug 2025)(opens in a new tab) | 2025-08 |
| GeForce RTX 5090 | LLaMA 2 7B (benchmark reference) | Q4_0 | Vulkan | 512 / 128 | 10381.6 | 263.6 | llama.cpp — Vulkan performance scoreboard(opens in a new tab) | 2025 → 2026 |
| Intel Arc B580 | LLaMA 2 7B (benchmark reference) | Q4_0 | Vulkan | 512 / 128 | 620.9 | 70.1 | llama.cpp — Vulkan performance scoreboard(opens in a new tab) | 2025 → 2026 |
| Radeon 8060S (40 CU) | Qwen3 30B-A3B Instruct 2507 | UD-Q4_K_XL (Qwen3-30B-A3B) | Vulkan | 512 / 128 | 755.1 | 85.1 | Strix Halo wiki — llama.cpp performance(opens in a new tab) | 2025 |
| Radeon 8060S (40 CU) | Qwen3 30B-A3B Instruct 2507 | UD-Q4_K_XL (Qwen3-30B-A3B) | ROCm (AMD) | 512 / 128 | 650.6 | 64.2 | Strix Halo wiki — llama.cpp performance(opens in a new tab) | 2025 |
| Radeon 8060S (40 CU) | LLaMA 2 7B (benchmark reference) | Q4_0 | Vulkan | 512 / 128 | 881.7 | 52.2 | llama.cpp — Vulkan performance scoreboard(opens in a new tab) | 2025 → 2026 |
| Radeon 8060S (40 CU) | LLaMA 2 7B (benchmark reference) | Q4_0 | ROCm (AMD) | 512 / 128 | 343.9 | 50.9 | llm-tracker — Strix Halo GPU performance (2025-05)(opens in a new tab) | 2025-05 |
| Radeon AI PRO R9700 | LLaMA 2 7B (benchmark reference) | Q4_0 | Vulkan | 512 / 128 | 5609.8 | 145.7 | llama.cpp — Vulkan performance scoreboard(opens in a new tab) | 2025 → 2026 |
| Radeon RX 7900 XT | gpt-oss-20b | MXFP4 | ROCm (AMD) | 2048 / 128 | 4251.6 | 101.9 | llama.cpp — gpt-oss guide & performance (Aug 2025)(opens in a new tab) | 2025-08 |
| Radeon RX 7900 XTX | LLaMA 2 7B (benchmark reference) | Q4_0 | Vulkan | 512 / 128 | 3727.0 | 182.6 | llama.cpp — Vulkan performance scoreboard(opens in a new tab) | 2025 → 2026 |
| Radeon RX 9070 XT | LLaMA 2 7B (benchmark reference) | Q4_0 | Vulkan | 512 / 128 | 5036.0 | 137.1 | llama.cpp — Vulkan performance scoreboard(opens in a new tab) | 2025 → 2026 |
| RTX PRO 6000 Blackwell | LLaMA 2 7B (benchmark reference) | Q4_0 | CUDA (NVIDIA) | 512 / 128 | 14854.6 | 274.2 | llama.cpp — CUDA performance scoreboard(opens in a new tab) | 2025-08 → 2026 |
| RTX PRO 6000 Blackwell | gpt-oss-120b | MXFP4 | CUDA (NVIDIA) | 2048 / 128 | 5518.1 | 196.3 | llama.cpp — gpt-oss guide & performance (Aug 2025)(opens in a new tab) | 2025-08 |