Ai2’s Olmo-core 3 Keeps Open MoE Training Efficient Past a Trillion Parameters

On October 1, 2026, the Allen Institute for AI released Olmo-core 3, a major upgrade to the open framework behind Olmo that rebuilds mixture-of-experts training so total parameter capacity can climb into the trillion-parameter range without collapsing throughput. The announcement lives on Ai2’s blog and the Hugging Face blog mirror; code remains under Apache-2.0 in allenai/OLMo-core, with a companion technical report covering systems design, ablations, and approaches Ai2 tried and discarded. This is infrastructure news, not a new chat checkpoint—but for labs that want to train sparse models instead of only downloading them, the stack is the story.

MoE models promise more learned capacity at a fixed active-parameter budget: only a few experts fire per token, so compute per step can stay manageable even as the expert pool grows. The catch is systems cost. The full set of experts still has to live across GPU memory, gradients still have to update them, and routing tokens to the right devices creates communication that can erase the theoretical win. Ai2’s illustrative scale-up holds active parameters near 3.2B while growing the expert pool from 8 to 128 and total capacity from about 4.6B to 47B, with training throughput falling less than 5%. Separately, the same infrastructure has been benchmarked above one trillion total parameters.

Olmo-core’s history explains the redesign. Sparse work goes back to OlmoE’s 64 routed experts; Olmo 3 itself was dense, so its training path assumed nearly all weights were active every step. The earlier MoE path in Olmo-core leaned on fully sharded data parallelism that gathered and reshared weights for each micro-batch. Olmo-core 3 switches toward distributed data parallelism that keeps experts resident on GPUs and moves the relevant data to them, avoiding repeated weight gathers. On eight NVIDIA B300 GPUs, Ai2 reports a 47B MoE processing about 52,000 tokens per second per GPU with the new stack versus about 19,400 with the older implementation—roughly 2.7×. Megatron-Core remains the established industrial alternative; Ai2’s claim is an integrated MoE stack inside the open Olmo tooling, not a monopoly on large-scale training.

Three distribution techniques carry the scale story. Expert parallelism shards the expert pool so each GPU holds only a slice. Pipeline parallelism splits layers across GPU groups to cut per-device memory. A distributed optimizer shards optimizer state instead of replicating it everywhere. Routing and compute optimizations sit on top: rowwise expert parallelism that writes routed tokens straight into expert input buffers, GPU-resident routing metadata so the CPU is not stalled on host copies, and grouped GEMM that batches many small expert matmuls into more efficient GPU work. MXFP8 support is optional lower precision; in a controlled four-B300 benchmark with uniform expert load, Ai2 measured about 21% higher end-to-end throughput versus BF16 and peak active memory down from 103 GiB to 95 GiB, with most gains in feed-forward and expert data movement rather than attention alone.

Trillion-parameter numbers need careful reading. Ai2 cites a 1.2-trillion-parameter configuration with 58.36B active per token across 512 B300 GPUs, peaking at 858 TFLOP/s/GPU under random routing used to stress the system rather than to report a trained model’s quality. A DeepEP v2 experiment reached 2.38 trillion total parameters as a short capacity test, not a full training run. The technical report also catalogs negative results that matter to anyone copying the stack: a load-balancing score that rose while real balance worsened (“token gerrymandering”), expert learning-rate heuristics that did not help the tested family, value-dependent GPU timings that break naive kernel comparisons, and communication-computation overlap that sometimes slowed end-to-end runs. Publishing those failures is part of Ai2’s open-science posture.

What Olmo-core 3 is not: a drop-in replacement for buying inference, a guarantee that the next Olmo will ship on day one of the blog post, or a claim that every lab can train a trillion-parameter MoE on a single node. It is the scaffolding Ai2 says it will use for a next-generation MoE Olmo aimed at larger data and longer context, opened so others can train their own sparse models, retarget hardware, and experiment with routing and parallelism. For researchers already in the Olmo orbit, the practical next steps are the GitHub tree, the report, and the interactive walkthrough Ai2 links for seeing data, expert, and pipeline parallelism compose from one GPU to many.

For Writeble’s opensource desk, Olmo-core 3 is the October 1 reminder that open weights matter more when the training system that can actually grow MoEs is open too.