Local AI Advisor

Inference runtimes

The runtime is the software that actually loads the model and runs it on your GPU or CPU; your AI tool talks to it over an API.

Compatibility matrix

Which GPU backends each runtime supports (best maturity on any OS) and which APIs it exposes to AI tools.
Runtimes by compute backend and API
RuntimeCompute backendsAPIs exposedFormats
MetalCUDAROCmVulkanSYCLCPUOpenAIAnthropicOllama
Ollamamaturematuregoodexperimental—goodGGUF
llama.cpp (llama-server)maturematuregoodgoodexperimentalmatureGGUF
MLX-LMmature—————MLX
LM Studiomaturematuregoodgood—goodGGUF, MLX
vLLMexperimentalmaturegood——experimentalsafetensors
SGLang—maturegood———safetensors
Jangoodgood—good—goodGGUF
LocalAIgoodgoodexperimentalgoodexperimentalgoodGGUF
Lemonade——goodgood—goodGGUF

Maturity: mature production-ready good works well, fewer usersexperimental expect rough edges. These are editorial assessments.

All runtimes

  • Ollama

    Model serverBeginner-friendly

    The easiest way to run local models. One-line install, model library, background server with Ollama, OpenAI- and Anthropic-compatible (v0.14+) APIs. Since v0.35 (Sep 2026) it also serves decision models through /v1/systemone (returns choices/probabilities/scores instead of text); since v0.40.4 (Oct 2026) decision models also work from the CLI (`ollama run --ask`) and on the MLX runner on Apple Silicon. Since v0.40 (Oct 2026), supported model architectures run on the MLX engine by default on Apple Silicon. Since v0.40.6 (Oct 2026) it also reads q2_0 GGUF files.

    Engine
    llama.cpp
    Model formats
    GGUF
    APIs
    Ollama, OpenAI, Anthropic
    Concurrency
    Moderate
    KV cache quantization
    Supported
    Local API server
    Yes

    Concurrency: Parallel slots work, but each extra request noticeably slows every stream. Prompt cache reuse: Fair — reused within a session, sometimes re-processed.

    Ollama: backend maturity per operating system
    BackendmacOSLinuxWindows
    Metal (Apple GPU)mature——
    CUDA (NVIDIA)—maturemature
    ROCm (AMD)—goodgood
    Vulkan—experimentalexperimental
    CPU onlygoodgoodgood
    Documentation(opens in a new tab)Ollama release notes — q2_0 GGUF support (v0.40.6)(opens in a new tab)verified 2026-10-17high confidence

    MLX-by-default confirmed; v0.40.6 (Oct 16) adds q2_0 GGUF tensor/file types and Clef cloud decision models. v0.40.4 added decision models to the CLI via --ask and decision-model support in the MLX runner. Decision-models API and Anthropic /v1/messages compatibility verified 2026-10-09.

  • llama.cpp (llama-server)

    Inference engineAdvanced users

    The reference GGUF engine. Widest hardware coverage, fine-grained control (offload, KV quantization, parallel slots), OpenAI- and Anthropic-compatible server. Since Oct 2026 (build 11361) it also serves decision models through /v1/systemone (choices/probabilities/scores in a single forward pass) and reports each model's input/output modalities in GET /v1/models.

    Engine
    llama.cpp
    Model formats
    GGUF
    APIs
    OpenAI, Anthropic
    Concurrency
    Moderate
    KV cache quantization
    Supported
    Local API server
    Yes

    Concurrency: Parallel slots work, but each extra request noticeably slows every stream. Prompt cache reuse: Good — follow-up turns usually skip re-reading the history.

    llama.cpp (llama-server): backend maturity per operating system
    BackendmacOSLinuxWindows
    Metal (Apple GPU)mature——
    CUDA (NVIDIA)—maturemature
    ROCm (AMD)—goodgood
    Vulkan—goodgood
    SYCL / oneAPI (Intel)—experimentalexperimental
    CPU onlymaturematuremature
    Documentation(opens in a new tab)ggml-org — New in llama.cpp: Decision Models (2026-10-02, /v1/systemone)(opens in a new tab)verified 2026-10-16high confidence

    Decision-models serving confirmed; Anthropic Messages API in llama.cpp (2026-01-19) verified earlier. v0.7.0 released 2026-10-15: LiLiCorr speculative decoding, MTP for GLM5-Next, GPU cache for MoE experts (up to ~2.2x decode speedup).

  • MLX-LM

    Inference engineIntermediate

    Apple's MLX framework with the mlx-lm package. Fast on Apple Silicon, uses M5 GPU Neural Accelerators, OpenAI-compatible mlx_lm.server.

    Engine
    mlx
    Model formats
    MLX
    APIs
    OpenAI
    Concurrency
    Mostly sequential
    KV cache quantization
    Supported
    Local API server
    Yes

    Concurrency: Concurrent requests largely queue up; best for one user at a time. Prompt cache reuse: Fair — reused within a session, sometimes re-processed.

    Hardware restriction: Apple Silicon Macs only.

    MLX-LM: backend maturity per operating system
    BackendmacOSLinuxWindows
    Metal (Apple GPU)· accel.mature——
    Documentation(opens in a new tab)mlx-lm server docs (not recommended for production; one request at a time with --kv-bits)(opens in a new tab)verified 2026-09-30high confidence

  • LM Studio

    Desktop appBeginner-friendly

    Desktop app with model browser, chat UI and local server. Runs GGUF via llama.cpp and MLX models on Macs; OpenAI- and Anthropic-compatible endpoints.

    Engine
    llama.cpp
    Model formats
    GGUF, MLX
    APIs
    OpenAI, Anthropic
    Concurrency
    Moderate
    KV cache quantization
    Supported
    Local API server
    Yes

    Concurrency: Parallel slots work, but each extra request noticeably slows every stream. Prompt cache reuse: Good — follow-up turns usually skip re-reading the history.

    LM Studio: backend maturity per operating system
    BackendmacOSLinuxWindows
    Metal (Apple GPU)· accel.mature——
    CUDA (NVIDIA)—maturemature
    ROCm (AMD)—goodgood
    Vulkan—goodgood
    CPU onlygoodgoodgood
    Documentation(opens in a new tab)LM Studio — Claude Code integration (v0.4.1+)(opens in a new tab)verified 2026-10-13high confidence

    v0.4.26+ ships /v1/systemone and /v1/decisions endpoints, CUDA 13 support on Windows ARM, and llama.cpp 2.50.0 runtime extension packs (official LM Studio changelog, Oct 2026).

  • vLLM

    Model serverAdvanced users

    High-throughput serving engine with PagedAttention and continuous batching. Best for many concurrent requests on NVIDIA/AMD GPUs under Linux.

    Engine
    vllm
    Model formats
    safetensors
    APIs
    OpenAI, Anthropic
    Concurrency
    Excellent batching
    KV cache quantization
    Supported
    Local API server
    Yes

    Concurrency: Continuous batching: parallel requests and agents barely slow each other down. Prompt cache reuse: Excellent — shared prompt prefixes are almost always reused.

    vLLM: backend maturity per operating system
    BackendmacOSLinuxWindows
    Metal (Apple GPU)experimental——
    CUDA (NVIDIA)—mature—
    ROCm (AMD)—good—
    CPU only—experimental—
    Documentation(opens in a new tab)vLLM repository (OpenAI + Anthropic Messages API); vllm-metal community plugin for Apple Silicon(opens in a new tab)verified 2026-09-30medium confidence

  • SGLang

    Model serverAdvanced users

    Serving engine with RadixAttention prefix caching — excellent for agents and many parallel requests sharing long prompts. Linux GPUs.

    Engine
    sglang
    Model formats
    safetensors
    APIs
    OpenAI
    Concurrency
    Excellent batching
    KV cache quantization
    Supported
    Local API server
    Yes

    Concurrency: Continuous batching: parallel requests and agents barely slow each other down. Prompt cache reuse: Excellent — shared prompt prefixes are almost always reused.

    SGLang: backend maturity per operating system
    BackendmacOSLinuxWindows
    CUDA (NVIDIA)—mature—
    ROCm (AMD)—good—
    Documentation(opens in a new tab)SGLang repository (platform list)(opens in a new tab)verified 2026-09-30high confidence

  • Jan

    Desktop appBeginner-friendly

    Open-source desktop chat app built on llama.cpp with a local OpenAI-compatible API server.

    Engine
    llama.cpp
    Model formats
    GGUF
    APIs
    OpenAI
    Concurrency
    Mostly sequential
    KV cache quantization
    Not supported
    Local API server
    Yes

    Concurrency: Concurrent requests largely queue up; best for one user at a time. Prompt cache reuse: Fair — reused within a session, sometimes re-processed.

    Jan: backend maturity per operating system
    BackendmacOSLinuxWindows
    Metal (Apple GPU)good——
    CUDA (NVIDIA)—goodgood
    Vulkan—goodgood
    CPU onlygoodgoodgood
    Documentation(opens in a new tab)Jan repository — OpenAI-compatible server on localhost:1337(opens in a new tab)verified 2026-09-30high confidence

  • LocalAI

    Model serverIntermediate

    Self-hosted OpenAI-compatible API (Docker-first) wrapping llama.cpp and other backends; good for homelab servers.

    Engine
    llama.cpp
    Model formats
    GGUF
    APIs
    OpenAI, Anthropic
    Concurrency
    Moderate
    KV cache quantization
    Supported
    Local API server
    Yes

    Concurrency: Parallel slots work, but each extra request noticeably slows every stream. Prompt cache reuse: Fair — reused within a session, sometimes re-processed.

    LocalAI: backend maturity per operating system
    BackendmacOSLinuxWindows
    Metal (Apple GPU)good——
    CUDA (NVIDIA)—goodgood
    ROCm (AMD)—experimental—
    Vulkan—goodgood
    SYCL / oneAPI (Intel)—experimental—
    CPU onlygoodgoodgood
    Documentation(opens in a new tab)LocalAI repository — OpenAI & Anthropic API compatibility(opens in a new tab)verified 2026-09-30high confidence

  • Lemonade

    Model serverBeginner-friendly

    AMD's local server for Ryzen AI and Radeon: manages llama.cpp (Vulkan/ROCm) plus NPU engines; OpenAI- and Anthropic-compatible APIs.

    Engine
    llama.cpp
    Model formats
    GGUF
    APIs
    OpenAI, Anthropic
    Concurrency
    Moderate
    KV cache quantization
    Supported
    Local API server
    Yes

    Concurrency: Parallel slots work, but each extra request noticeably slows every stream. Prompt cache reuse: Good — follow-up turns usually skip re-reading the history.

    Hardware restriction: AMD hardware only.

    Lemonade: backend maturity per operating system
    BackendmacOSLinuxWindows
    ROCm (AMD)—goodgood
    Vulkan—goodgood
    CPU only—goodgood
    Documentation(opens in a new tab)Lemonade — Anthropic API(opens in a new tab)verified 2026-09-30high confidence