NVIDIA Opens Four 16B Cosmos 3 Nano Checkpoints for Interactive Driving and Bimanual Robot Worlds

NVIDIA has released four open-weight Cosmos 3 Nano post-trained checkpoints aimed at interactive world simulation for autonomous driving and two-arm robots, each carrying 16 billion trainable parameters under the OpenMDW1.1 license. The Hugging Face blog post dated November 5, 2026 frames the drop as an open foundation for robotics and AV teams that need controllable synthetic video and sensor streams rather than another chat model. All four checkpoints are commercial- and non-commercial-usable under OpenMDW1.1’s terms. The published stack expects Linux, an NVIDIA GPU, Docker, and the NVIDIA Container Toolkit; NVIDIA notes that vLLM-Omni was tested on one GB200, without publishing a minimum-memory floor.

The four named variants split by job. Cosmos3-Nano-Transfer-Auto targets offline synthetic driving: up to 11 camera views including fisheyes, joint RGB–LiDAR, World Scenario Map (WSM) controls, and text prompts for appearance. Cosmos3-Nano-Sim-Auto is the causally distilled streaming sibling—few denoising steps, successive chunks of synced camera and LiDAR—and needs a simulation runtime plus a WSM renderer for closed-loop work. Cosmos3-Nano-Sim-Transfer takes simulator spatial controls (depth, segmentation, edges, blur) plus text and emits photoreal RGB while leaving physics inside the simulator. Cosmos3-Nano-Sim-Bimanual maps actions to video for two-arm platforms such as YAM and AgiBot, driven by joystick or policy; a mixed-action example uses 59 controls padded to 64. Together they cover offline AV synthetic data, streaming closed-loop AV sim, controllable photoreal transfer, and dual-arm action video.

Mixture-of-Transformers backbone and a four-stage student

Under the hood the models use a Mixture-of-Transformers design: separate text and visual networks joined through attention. Streaming follows a generate-decode-update loop—controls and history produce a chunk, the chunk is decoded, and history is updated for the next step. After the initial image, each latent step corresponds to four RGB frames. Training is staged: Stage0 bidirectional adaptation, Stage1 teacher forcing, Stage2 consistency distillation down to four denoising steps, and Stage3 self-forcing on the student’s own history. That distillation path is how Sim-Auto stays interactive with few steps instead of a slow offline sampler. NVIDIA also documents post-training recipes that start from the Cosmos3-Nano base so labs can adapt sensor rigs and embodiments without rebuilding the foundation from scratch.

What “interactive” means in practice

Interactive world simulation here is not a single video clip from a text prompt. Transfer-Auto can synthesize multi-view driving scenes with LiDAR under WSM and appearance text. Sim-Auto turns that into a streaming loop for closed-loop evaluation once a runtime and WSM renderer are attached. Sim-Transfer keeps the physics engine authoritative and only asks the model for photoreal RGB conditioned on spatial maps and text—useful when teams already trust their simulator dynamics but want better visuals. Sim-Bimanual closes the loop for dual-arm workcells: action vectors in, video out, with padding to a fixed control width so mixed joystick and policy inputs share one interface. The blog’s vision statement is blunt: an open foundation for interactive world simulation aimed at robotics and autonomous driving, not a general multimodal chat stack.

Hardware and serving caveats are part of the release, not footnotes. The containerized path assumes NVIDIA GPUs and the Container Toolkit; the only published serving data point is vLLM-Omni on a single GB200. Teams without that class of hardware will need to treat the cards as research artifacts until lighter footprints or clearer memory guidance appear. License-wise, OpenMDW1.1 is the gate for both commercial and non-commercial use under its published terms—simpler geography than some recent large-model licenses, but still a license to read before shipping product video.

For AV and robotics labs already deep in synthetic data pipelines, the concrete offer is four specialized 16B Nano checkpoints, shared Mixture-of-Transformers streaming mechanics, a four-stage distillation recipe that lands at four denoising steps, and post-training hooks from the Cosmos3-Nano base for new sensor layouts and embodiments. Offline multi-camera driving, closed-loop streaming AV, physics-preserving photoreal transfer, and bimanual action video are the four doors; which one matters depends on whether the bottleneck is data generation, closed-loop eval, visual fidelity on top of an existing sim, or dual-arm policy visualization.