Magnitude Tunes On-Device Kernels So Local Agents Outrun Stock llama.cpp

Magnitude is shipping as an Apache-2.0 local inference engine built for agent workloads: it profiles the machine you are on, compiles and tunes kernels for that exact chip, then exposes open-weight models to the coding and personal agents people already run. The product surface is a desktop app for macOS, Windows, and Linux with a bundled magnitude CLI; the code lives at magnitudedev/magnitude, and the company site documents connections to Pi, OpenCode, Hermes, OpenClaw, Codex, Claude Code, Oh My Pi, and Cline, plus an OpenAI-compatible API for everything else. In the early October 2026 product window, the hardware story widened to Apple Silicon, NVIDIA, AMD GPUs, and CPU-only boxes without a fixed minimum—smaller machines simply run smaller models.

The performance claim is specific enough to evaluate. Against llama.cpp on Qwen 3.6 35B-A3B at 4-bit with 64k context and no speculative decoding, Magnitude publishes Metal numbers on a Mac M4 Pro 48 GB of about 9% faster prefill (466→507 tok/s) and 92% faster decode (30→57 tok/s), and CUDA numbers on a DGX Spark of about 23% faster prefill (2,033→2,507 tok/s) and 19% faster decode (49→58 tok/s). Marketing copy rounds that to “up to 2× faster than llama.cpp.” Those are vendor benchmarks on named configs; teams should re-run on their own SKUs before rewriting capacity plans. The design rationale is still clear: generalist engines ship precompiled kernels for broad hardware classes, while Magnitude tunes on the device before a model runs so the kernels match the chip under the desk.

Agent-shaped inference, not chatbot demos, drives the memory story. Magnitude says it allocates only what is needed for weights at first, grows session memory as context lengthens, and releases memory when an agent stops—citing about 27% less memory per agent in its materials. Concurrent sessions share a prefix cache for repeated system prompts and tool schemas, which is exactly the traffic pattern multi-agent desktops generate when several tools sit in the same long preamble. Privacy is framed as local-first: once weights are downloaded, prompts and files need not leave the machine. That places Magnitude beside Ollama and LM Studio in the “run open models at home” category, but with agent connection UX and autotuning as the differentiators rather than a model catalog alone.

Install flow is deliberately short. Download the app, pick a recommended model in Discover after hardware detection, connect an agent in Connections, and work. Hand-optimized kernels for popular open-weight families are how Magnitude explains beating generalist engines; the supported model list is maintained at magnitude.dev/models rather than promised as universal GGUF compatibility. For harnesses outside the one-click list, the OpenAI-compatible local API is the escape hatch—useful for custom Strands, OpenHands, or in-house runners that already speak that protocol.

Where Magnitude fits relative to adjacent October stories matters for editors. It is not a harness like Lambda’s Open Jarvis write-up (which retargets agent specs around a local model) and not a decision-model checkpoint like Clef or Strands Decider. It is the inference runtime underneath those loops: if your agent stack already chose Qwen or another open family, Magnitude argues you can keep the agent and replace the slow local backend. It also is not a cloud vLLM replacement; there is no pitch here for multi-node serving, only for private machines that want agent-grade decode without token bills.

Caveats stay in view. Speedups vary sharply by backend—Metal decode gains dwarf the CUDA decode gains in the published charts—so NVIDIA workstation users should not assume the 2× headline. Kernel coverage is family-specific; obscure architectures may fall back or simply be unsupported. Apache-2.0 openness covers the project’s stated license on the site and LICENSE file, but production teams still need their own eval harness for latency, quality, and memory under real tool-calling traces. For developers already juggling Pi, Hermes, or Claude Code against a lukewarm local backend, the October signal is straightforward: try the autotuned engine before buying another GPU solely for decode headroom.

For Writeble’s products desk, Magnitude is a sharp local-inference product bet—device-tuned kernels and agent connectors, not another foundation-model launch.