Underdog Saluki 27B Squeezes Qwen3.8-27B Into 7.89 GB and Beats the Full Model on Tool Calls

Underdog, the on-device assistant built by Conway Research, has released Saluki 27B 1.0, a compressed build of Alibaba’s Qwen3.8-27B that fits in 7.89 GB and, on the company’s own tool-calling test, outperforms the full 54 GB model it came from. The file is a standard GGUF that runs in unmodified llama.cpp and the apps built on it. It is now on Hugging Face under the ConwayResearch/Underdog-Saluki-27B-1.0 repository with an Apache 2.0 license.

Shrinking a model to about two bits per weight usually costs accuracy across the board. Underdog’s bet was to choose which abilities to protect. The team tuned the compression to preserve tool calling—the skill that lets a chat model act as an agent by emitting structured function calls—because that is what an assistant running on a laptop needs most.

The file and what it needs

Saluki is an IQ2-mix quantization built on Qwen3.8-27B from the Qwen team and on ISTA-DASLab’s Qwen3.8-27B-GSQ-RCO GGUF, both Apache 2.0 projects credited in the repository notice. The main file is text-only. An optional vision projector, 928 MB at F16 or 629 MB at Q8_0, adds image input and is passed to llama.cpp alongside the main file. Thinking mode is on by default and can be switched per request, as with the base model.

The model card’s server example loads all layers to the GPU with flash attention and a 32,768-token context, exposing an OpenAI-compatible chat API. A template flag in that command activates the Qwen3.8 chat template, which handles tool calls and thinking. For general use the card recommends thinking on with temperature 0.6, top_p 0.95 and top_k 20; for fast, direct tool calls it suggests thinking off at temperature 0. Fitting a 27B-class model in under 8 GB puts it within reach of a single consumer GPU. On an RTX 3080 with 12 GB, at 32K context with an 8-bit KV cache, the original Saluki file used 10.3 GB of VRAM and generated about 41 tokens per second.

A tool-calling test frozen in advance

Underdog Bench is 120 tasks drawn from the Berkeley Function Calling Leaderboard v4, frozen before any model was tested and run with thinking off at temperature zero. Saluki passed 88. The full-size Qwen3.8-27B, run by Underdog on the same harness, passed 84. PrismML’s Bonsai 2 27B, a 5.95 GB file, passed 70. On 100 parallel tool-call tasks, where a model must issue several calls in one reply, Saluki scored 42 against the full model’s 35.

Underdog reports average retention of about 96% across nine benchmarks. Instruction following held up: IFEval prompt-loose came in at 93.5 against a public 91.5 for the full model, and IFBench at 72.7 against 71.0. On SWE-bench Verified, measured as issues fixed out of 50, Saluki solved 30 versus 33.

The trade it makes

The losses concentrate in reasoning. Saluki keeps about 82 to 85% of the full model’s competition-math score; on AIME 2025 averaged over four runs it posted 79.2 against a public 96.7 for the original, though the card notes those figures come from different harnesses. On AIME 2026 the Saluki card lists a similar gap, 80.0 against 94.6 for the full model. MBPP+ fell from 83.9 to 78.0 and MuSR from 79.6 to 67.5.

The model card lists other weak spots plainly. Saluki struggles with letter-level puzzles such as palindromes, vowel rules and alphabetical ordering. About a fifth of its parallel-call replies contain small formatting slips. With thinking on, it often reasons at length before answering, which costs time on local hardware.

Why a small assistant company is shipping weights

For Underdog, the release showcases the model behind a local-first assistant. For developers, it is a ready-made option for agents that must keep data on the device: a model that calls tools reliably, runs in the most widely deployed local runtime, and carries a license that permits commercial use.

The community has already started building on it. A derivative repository, Kujira/Underdog-Saluki-27B-1.0-MTP-GGUF, grafts Qwen3.8-27B’s own multi-token-prediction head onto the unchanged Saluki weights so llama.cpp can use self-speculative decoding. The 8.35 GB file adds 15 tensors, about 0.45 GB, while all 851 original tensors stay byte-identical; nothing was retrained or re-quantized. On that same RTX 3080, generation rose from 41.2 to 66.2 tokens per second on average, with VRAM at 11.5 GB. Its author notes the output is not bit-identical to Saluki at temperature zero, advises keeping 12 GB cards at 32K context, and reports no new quality benchmarks.

Instead of chasing the smallest file or the best average score, Underdog picked one capability that matters for agents and measured it on a test fixed before tuning began. Teams evaluating Saluki should run their own tool schemas through it, and keep a larger model in reserve for math-heavy or long-reasoning work.