NVIDIA Cosmos Expands Tools for Training and Deploying World Models

NVIDIA has expanded its Cosmos platform to help developers rapidly build world models, which enable physical AI systems to understand and simulate the real world. The new tools include training data preparation, simulation checkpoints, and deployment recipes.

The NVIDIA Cosmos platform is already ranking top five on six different open model leaderboards, giving world model builders access to frontier open models, as well as data, training, and optimization tools and recipes. This allows developers to scale training data, adapt models, test behavior, and optimize for performance before deployment.

Building world models begins with multimodal, multisensory data. NVIDIA Cosmos Data Factory brings together the necessary tools and skills for labeling, curating, augmenting, and evaluating data at scale for training physical AI. This includes GPU-accelerated solutions like Cosmos Curator, which has now added support for multimodal multisensory inputs and processing blocks.

Cosmos Data Factory offers five cloud-native blueprints that are available through GitHub, Microsoft Azure, Nebius, and CoreWeave. These blueprints include Autonomous Vehicle Dataset Generation, Image Attribute Augmentation, Event Video Generation, Defect Image Generation, and Video Data Augmentation. They accelerate data curation, processing, and synthetic data generation with agentic skills.

Companies are already putting Cosmos Data Factory blueprints to work to speed their physical AI development workflows. Porsche is leading the development of a shared AV data flywheel across Volkswagen Group brands using Cosmos Data Factory through Voxel51 on Microsoft Azure. This is expected to reduce repeated physical test drives by up to 90% and shorten development cycle times by up to 85%.

Porsche is searching and augmenting autonomous vehicle data footage in Voxel51, powered by Cosmos Data Factory. ANYbotics generated over 1,500 synthetic video clips with Cosmos Data Factory blueprints, curating real-world data and validating outputs with Encord’s platform, to train its industrial-inspection robots to recognize liquid spills and other changes between facility visits.

Ultralytics, a computer vision company, retrained its YOLO model with real images and targeted synthetic defects generated using Cosmos Data Factory. This improved defect localization and recall. With training data prepared, developers can now train their own architecture or start with a pre-trained model.

NVIDIA extended the Cosmos Framework training, optimization, and deployment tools and recipes as agent skills to more open models at GTC Berlin. Developers and agents can adapt open models like GR00T, Alpamayo, QWEN, Pi0.5, and other compatible Hugging Face models through fine-tuning and reinforcement learning, optimize the model with distillation and quantization, and deploy with high-performing inference.

To develop a world model specialized for egocentric manipulation, robotics research lab MicroAGI post-trained Cosmos 3 on videos of humans manipulating objects. It distilled its 32-billion-parameter reasoner down to an 8-billion-parameter Nano model with no loss in accuracy. Developers can label data and post-train Cosmos 3 on Roboflow for their specific environments and tasks.

NVIDIA continues advancing foundational technology through its work on Cosmos models. New Cosmos 3 Nano model variants for simulation enable developers to test autonomous policies in repeatable, hard-to-capture, multiview multisensor scenarios. Accompanying code allows developers to adapt the models to train and evaluate their own policies for autonomous vehicles and bimanual robots in interactive simulations before real-world testing.

Developers can deploy core Cosmos models using updated NVIDIA Cosmos 3 NIM microservices, which deliver up to 2x faster inference with no quality loss in validated configurations. Continuous video analysis enables sub-second alerts. For vision AI developers, the new NVIDIA Metropolis Blueprint for video search and summarization (VSS) 3.3 turns one prompt into an efficient video AI application.