Qwen-Audio-3.1 Realtime Trains Voice Agents to Think, Act, and Choose When to Speak

Alibaba’s Qwen team released Qwen-Audio-3.1-Realtime, a full-duplex speech model built for voice agents that reason, call tools, and manage turn-taking—available on QwenCloud as qwen-audio-3.1-realtime-plus over WebSocket, with the technical report “Towards Reliable Agentic Voice Interaction” on arXiv as 2609.25176. No open weights were announced; access is through the managed API. The model page lists a 262,144-token context window (245K max input, 16K max output), function calling, web search, structured outputs, context cache, and fine-tuning. List pricing on the product page is $6.4 per 1M audio input tokens, $0.8 per 1M text input tokens, and $24 per 1M tokens for text-and-audio output (output text not charged separately). Default limits include 60 requests and 100K tokens per minute.

Version 3.0 already covered real-time listening and speaking. Version 3.1 asks what changes when spoken interaction becomes a setting for reasoning and action: retain constraints as requests evolve, choose and execute tools, interpret feedback, and decide whether to speak. The report organizes the system as Think, Act, and Speak and Coordinate. Runtime architecture runs two models that share the same Audio Encoder plus LLM design. A full-duplex decision model predicts whether to keep listening, speak, stop, or resume. A speech-to-text model writes response content as text. A context-aware voice renderer then turns that text into streaming speech conditioned on conversation history, voice cues, and acoustic context.

Think is foundation post-training. Core-Cocktail supervised fine-tuning re-anchors the audio model to its source text LLM on million-hour-scale paired data. Multimodality on-policy distillation follows: a Text Teacher and a frozen Audio Reference score each token of the student’s own trajectory rather than imitating pre-written answers. Domain experts for empathy, pragmatic intent, and acoustic scenes train with Group Relative Policy Optimization (GRPO); Multi-Teacher OPD merges them into one deployable model. The report groups the two OPD stages as M²-OPD.

Act is where agentic behavior is trained. Each executable domain bundles a tool pool, a stateful JSON database, and a natural-language business policy, seeded from open-source tool and MCP server definitions. Tasks define one of three terminal outcomes: a write, a justified refusal, or an unsupported request. Scoring checks terminal state, then permitted writes, then behavioral assertions—so a fluent reply cannot rescue a failed state check. GRPO receives rewards at dialogue, milestone, and turn levels. Search training penalizes redundant queries; the report says mean queries per search call fell from 4.37 to 1.05 while trigger F1 slipped from 60.87% to 58.61% on an in-house WebSearch1K setting.

Speak and Coordinate decides whether, when, and how to speak. On Full-Duplex-Bench v1.5, responses to background speech fall from 0.73 to 0.13, and replies when the user is talking to someone else fall from 0.13 to 0.03. On FDB v3.0, filler rate drops from 0.7590 to 0.2960. Trade-offs remain: after interruptions, unwanted resume rate rises from 0.035 to 0.130, and interruption stop latency is 1.116 seconds versus 0.383 for GPT-Realtime-2 in the report’s table. On a half-duplex speech-to-text adaptation of τ-Voice, overall task success rises from 78.4% to 82.0% versus 3.0 (not comparable to official full-duplex τ-Voice). Audio MultiChallenge rises from 47.12 to 52.21; a 14-language Big Bench Audio average climbs from 81.7% to 88.1%; FLEURS WER falls from 9.01 to 3.98. Multi-turn attack success falls to 26.0% (Chinese) and 23.5% (English) in the report’s in-house sets. A 50-session human red-team study still has GPT-Realtime-2 ahead on session-level safety pass rate (96.00% versus 92.00%).

A separate Voice Harness prototype—evaluated with a 3.0 foreground, not as a 3.1 model benchmark—shows foreground–background coordination for persistent spoken tasks with bounded memory. That system story is explicitly carved out from the 3.1 model evaluation. For builders, the QwenCloud page ships a WebSocket sample against wss://maas.qwencloudapi.com/api-ws/v1/realtime?model=qwen-audio-3.1-realtime-plus with server VAD turn detection and streaming audio deltas.

For Writeble’s voice-ai desk, Qwen-Audio-3.1 matters because it treats duplex timing, tool-verified outcomes, and refusal boundaries as first-class training targets—not just lower latency speech.