Yandex Ungates AliceAI Foundation 80B as an Apache MoE With 262K Context

Yandex published AliceAI-Foundation-80B-A3B-Base on Hugging Face as an ungated Apache 2.0 base model and detailed the release in a September 21, 2026 company post on Habr. The English model card describes an 80-billion-parameter hybrid architecture with mixture-of-experts layers that activate about 3 billion parameters per token, a 262,144-token context window, and a training story that starts from scratch rather than continued pretraining on a borrowed checkpoint. Hugging Face API metadata lists the repository as ungated, license: apache-2.0, created September 12 and last modified September 22, 2026. The LICENSE file is Copyright 2026 YANDEX LLC under Apache License Version 2.0.

Architecturally the card is unusually specific for a public base drop. The model is an autoregressive language model at the pretraining stage with hidden size 2048, vocabulary 129,024, and 48 layers arranged as twelve repeats of a block that stacks three KDA-to-MoE stages and one gated-attention-to-MoE stage. KDA layers use 32 query and 32 KV heads at dimension 128 with a convolution kernel of size 4. Gated attention uses 16 query heads and 2 KV heads at dimension 256. Each MoE block has 512 experts, routes top-10 plus one shared expert, and uses expert intermediate size 512. A single MTP layer is listed alongside the long context ceiling. Those details matter for people wiring Transformers or vLLM: the card pins Transformers 5.16.1, accelerate 1.14.0, and flash-linear-attention 0.5.0 with KDA support, and it requires trust_remote_code=True when loading.

Yandex says key design choices were validated with separate from-scratch runs of about 2 trillion tokens each while the team rebuilt the corpus, architecture, and data mixture for reasoning and tool use. The English README is careful about what the weights are: a pretrained base, not an aligned chat product. It ships without a default chat_template in tokenizer_config.json precisely so inference does not pretend there is one conversational format. For people who will fine-tune, the repository includes an OpenAI Messages–style trajectory format, a Jinja template used during pretraining for roles, reasoning traces, and tool calls, a minimal PEFT LoRA example under FSDP2 aimed at four 80 GB GPUs, and a warning that production use needs the downstream team’s own post-training and behavioral controls.

On the evaluation tables Yandex publishes from internal vLLM harnesses, the model is positioned as competitive with larger open bases on math and coding while leading several Russian factual and educational benches the company also released. WikiWebFacts and HardMultiQA ship as Hugging Face datasets with evaluation protocols; EduBench splits cover Russian, literature, history, English, and math drawn from Alice assistant traffic. Headline numbers on the card include WikiWebFacts 86.5, HardMultiQA 67.9, MATH-500 91.1, and strong pass@k results on AIME 2026 and HMMT 2026 February under the company’s sampling settings. English TriviaQA and some coding rows trail other open bases in the same table, which the card leaves visible rather than marketing away.

Serving paths are documented for both Transformers and a Dockerized vLLM image (yamlbrand/alice-ai-vllm) with tensor parallel examples on four GPUs, FlashAttention v2, and optional MTP speculative decoding. The tokenizer loads as a SentencePiece BPE LlamaTokenizer; chain-of-thought and tool markers are ordinary vocabulary tokens, and legacy=false is fixed so whitespace behavior matches training.

Operationally, the base-model framing is a feature for labs that want to own post-training. Because AliceAI-Foundation-80B-A3B-Base is pretrained rather than instruction-tuned, teams can apply their own SFT and RL stacks without fighting an opaque chat prior, provided they render trajectories with the included template so tool and reasoning markers match pretraining. The card’s LoRA recipe is modest on purpose: it trains only on responses, saves only the adapter, and documents FSDP2 flags for a four-GPU box—enough to validate the load path before a larger run. Combined with the Apache license and ungated download, that lowers the barrier relative to gated or research-only MoE bases that still require access requests before a first from_pretrained.

For Writeble’s opensource desk, AliceAI Foundation 80B clears the bar that hosted SKU announcements do not: ungated weights, a permissive Apache license, verified dates, and enough architecture plus eval protocol to reproduce a load path. It also lands cleanly after the September 22–23 package without colliding with MiMo-V2.6 or DeepSeek Flash coverage. Teams that need a commercially friendly long-context MoE base—especially with Russian factual strength and an explicit tool-use pretraining story—now have a named checkpoint, LICENSE, and fine-tune recipe to evaluate on their own hardware rather than another waitlist.