Developers Put Frontier AI Models to Work with NVIDIA Omniverse Libraries

Frontier AI models are teaming up with NVIDIA’s Omniverse libraries in a big way. Together, they’re making it easier than ever to turn simulation ideas into working applications.

When developers use these tools together, the process of building simulations for exploring scenarios, investigating failures, and improving designs becomes much more efficient.

In one project, developers are using AI agents to direct them through natural-language instructions. This helps them review results and guide changes as needed.

The Omniverse libraries provide GPU-accelerated physics, rendering, and sensor simulation capabilities that help carry out the work quickly.

One key project showcasing this collaboration is a humanoid simulator for a warehouse environment. Frank DeLise, an NVIDIA product manager, used Astra to turn a SimReady warehouse and humanoid robot into an interactive simulator with first- and third-person views.

DeLise directed Astra to connect various NVIDIA Omniverse libraries, including ovphysx for physics, ovstage for scene updates, ovrtx for rendering, and ovui for the user interface. He also used Astra with SimReady to create a physical scene in simulation.

The resulting simulator allows developers to explore task behavior and evaluate how work gets done before automating warehouse tasks. This interactive environment is essential for evaluating different approaches and making informed decisions.

Another project demonstrating this collaboration’s power is an autonomous-driving testing workflow. Doyub Kim, a manager on NVIDIA’s simulation technology team, used Astra to build Zero to Alpamayo – a reusable simulation environment based on San Francisco’s Market Street.

Kim directed Astra to map out the workflow and connect asset creation, traffic, Omniverse RTX sensor simulation, and Alpamayo driving in stages. The resulting prototype served as a testing ground for comparing models and tracing how scene or sensor changes affect downstream driving behavior.

This collaboration also enables developers to create digital twins by comparing recorded and simulated camera and lidar outputs. Ashley Reid directed Astra and Claude Fable 5 agents to compare ovrtx camera and raw LiDAR outputs with recorded data.

Reid guided an iterative workflow in which the agents measured differences, created or modified OpenUSD scenes, and checked the results. Changes addressed missing objects, geometry, and materials, with acceptance depending on camera and lidar metrics.

In this process, developers use measured discrepancies to guide scene creation and improvement. This helps them build better simulations more efficiently.

Tae Kim, who leads NVIDIA Omniverse engineering and product, used sports videos and natural-language instructions to guide Astra in building Robo Olympics – an experimental project that tests simulated Unitree G1 humanoids performing sports movements.

Under Kim’s direction, Astra built controllers and refined them through physics trials. Newton Physics Engine simulated behavior, the open-source NVIDIA Warp framework accelerated calculations, and ovrtx rendered scenes and virtual-camera images.

In one experiment, the robot cleared a single hurdle in 64 of 100 simulation trials. This shows how Astra can help developers refine their simulations more accurately.

This collaboration also enables developers to test robotic disassembly with computer-aided design and simulation. Jens Jebens directed Astra to model a car suspension in PTC Onshape and configure it in NVIDIA Isaac Sim.

With Astra’s help, Jebens explored computer-aided design and tooling revisions for robots informed by simulation. The agent measured the available space and designed a wrench that the robot could use to reach the suspension’s bolts.

Jebens reported successful removal of a suspension component in simulation. This kind of testing is crucial for ensuring robotic systems can disassemble objects safely and efficiently.

Nic Johns prompted Astra to assemble NASA assets into an OpenUSD International Space Station model with telemetry. Johns built the application with a single prompt, then used a follow-up prompt to shift the scene to Earth’s daytime side so that the planet was visible.

The workflow used Blender for asset preparation and Omniverse libraries for rendering (ovrtx), scene runtime (ovstage), and streaming (ovstream). The application brings 3D models and operational data into the browser, with Johns guiding its development through prompts and corrections.

Chirag Majithia directed Astra to turn stereo camera captures into an editable OpenUSD studio. The workflow combined PyCuSFM, FoundationStereo, and nvblox for reconstruction, with user review guiding object selection and placement.