NaiveAI’s 309B Flash Model Drops Full Attention for a Million-Token Coding Stack

NaiveAI published “Naive-N0.5-Flash: Building Frontier AI with AI” on September 27, 2026, releasing an open-weight 309-billion-parameter mixture-of-experts model with 15.5 billion active parameters per token under the MIT License. The Beijing lab’s pitch is architectural and process-driven at once. The network uses a hybrid of Sliding-Window Attention and DeepSeek Sparse Attention with no full-attention layers anywhere, targeting a native one-million-token context for coding and AI research workloads. Separately, NaiveAI argues the model was developed inside an AI-centered R&D system in which models write code, run experiments, and propose iterations while humans set objectives, constraints, and decision gates.

Model-card numbers line up with the research post. The transformer is 48 layers arranged as eight six-layer modules. A standard module stacks five SWA layers and one DSA layer; the first layer of the first module is also DSA. SWA uses a 128-token window. DSA selects the top 2,048 tokens for backbone attention after a lightweight indexer scans history. Grouped-query attention uses four KV groups, and the DSA indexer runs 16 query heads—NaiveAI says that co-designed indexer cut index-selection wall time by 44% versus the original DSA setup while preserving agent-task performance at million-token context. The base is Xiaomi’s open MiMo-V2.5, which originally mixed SWA with a smaller set of global-attention layers that became costly at long context; Naive-N0.5-Flash replaces those global layers with DSA.

Training after the architecture change is described as 3.25 trillion tokens at native 1M context: 50B tokens of indexer warmup with other parameters frozen and a KL objective aligning the indexer to full-attention references; 3T tokens of sparse-attention continued pretraining focused on AI R&D and coding; and 200B tokens of learning-rate decay under supervised fine-tuning. Hybrid sequence parallelism is called out explicitly—Ulysses-style all-to-all for DSA’s full-history needs, and halo exchange limited to the SWA window for local layers—so communication cost for SWA drops from O(L) to O(w). NaiveAI says the resulting training system can process about one trillion tokens in roughly four days on 512 GPUs at 1M sequence length.

Inference is the second product surface. NaiveRT, the team’s runtime for this model, is built for single-stream decode speed with bitwise-deterministic sampling—important for RL rollouts whose generated tokens dominate wall-clock even when most of a million-token context is ingested observations. On eight GPUs, NaiveAI reports a peak single-stream decode of 2,122 tokens per second over the best one-second window across 41 HTML/SVG generation requests (thinking off, temperature 0.4, top-p 0.95, excluding prefill). A full speculative-decoding round is listed at 3.4 ms versus 12.3 ms for SGLang on the same system—a 72.4% reduction. Serving claims include about 50 tokens/s per user in Standard mode and up to 2,000 tokens/s in Ultrafast mode. API pricing, when access ships, is stated at $0.10 / $0.40 / $0.01 per million tokens for input, output, and cache reads. NaiveRT source, Docker Compose, and W8A8 plus DFlash draft weights are pointed to GitHub and Hugging Face, with some runtime materials marked as arriving by October 12.

The AI-centered R&D narrative is backed with engineering case studies rather than slogans alone. NaiveRT’s core path was built in six days across 151 documented optimization trials (63 adopted, 71 rolled back, 17 exploratory), with humans defining acceptance criteria and models profiling, implementing, and validating candidates—including rejecting a MoE fusion path that was numerically correct but slower end-to-end because PDL already overlapped kernel boundaries. A separate AutoWM world-model run tasked Naive-N0.5-Flash with research outside the lab’s prior expertise; after 400 hours and 15 major experimental rounds, AutoWM scored 77.43 on WorldArena-1 Track 1 under the public protocol, above a prior published high of 73.64 on that track as cited by NaiveAI.

Evaluation tables compare coding and agent harnesses such as DeepSWE, Terminal-Bench, ALE-CLI, and others, mostly inside Claude Code 2.1.207 with basic file and Bash tools. Those scores are company-reported and often sit beside competitor numbers taken from other vendors’ own harnesses, so readers should treat cross-model gaps as directional. Weights and inference code remain MIT-licensed on Hugging Face (NaiveAI/Naive-N0.5-Flash) with the technical blog as the primary narrative.

For Writeble’s opensource desk, Naive-N0.5-Flash is the late-September MIT MoE that makes “no full attention at 1M context” a concrete hybrid stack—and ties the release to a documented AI-assisted systems loop instead of a weights-only drop.