NVIDIA: Cosmos World Foundation Model Platform for Physical AI

Summary of https://d1qx31qr3h6wln.cloudfront.net/publications/NVIDIA%20Cosmos_3.pdf

Introduces NVIDIA's Cosmos World Foundation Model (WFM) platform for Physical AI. Cosmos uses a pre-training and post-training paradigm, employing both diffusion and autoregressive models trained on a massive, curated video dataset (20M hours) to create generalist WFMs.

These are then fine-tuned for specialized Physical AI tasks like robotic manipulation and autonomous driving. The platform includes a novel video tokenizer for efficient processing and a guardrail system for safety.

Results demonstrate state-of-the-art performance across various benchmarks and applications.