Skip to main content
Use casesAdvanced6 min read

NVIDIA's Cosmos3-H-Dreams: real-time surgical simulation, and local robotics

i
iaLocale Team
•
NVIDIA's Cosmos3-H-Dreams: real-time surgical simulation, and local robotics

NVIDIA has released Cosmos3-H-Dreams: a surgical world model that, on a single workstation GPU, generates in real time the endoscopic video of a robotic cholecystectomy, from the commands of the robot's two arms. It is not a language model: it does not answer questions or write code. It is an action-conditioned video generator — and that difference changes what kind of questions it can answer.

What it actually is, and how it is built

Cosmos3-H-Dreams is a real-time, action-conditioned world model trained for the CMR Surgical Versius robotic system. Given a single endoscopic camera frame, the recent history of robot commands, and the stream of future commands for both arms, it autoregressively generates the resulting endoscopic video: what the operating room screen would show if the robot executed those movements.

The architecture is a Mixture-of-Transformers with about 15.17 billion parameters: a Qwen3-VL-8B-Instruct reasoning tower combined with a latent-diffusion generation expert, sharing multimodal attention and operating on the latents of the Wan2.2 video VAE (4x temporal, 16x16 spatial compression).

The data is real: 4,792 episodes of recorded robotic cholecystectomy, roughly 17 million frames at 60 Hz and 1080p, resampled to 10 Hz. Fine-tuning used a strict selection: 64 recordings, about 526 unique minutes, with exactly two instruments visible in every window — the classic grasper (left arm) and hook (right arm) pair.

The key piece is distillation. A bidirectional teacher that sees the whole sequence and takes 35 sampling steps per frame is slow by definition. NVIDIA distilled it into a causal student that generates the video window by window with 4 steps per frame, using a KV cache — and that is the model that runs in real time.

How it runs at inference

At inference the model takes a starting frame and, optionally, up to 16 steps of action history (1.6 seconds). Each step is a 28-dimensional action vector: 14 per arm, covering anchor-relative translation and rotation, jaw opening, electrosurgical energy, and joystick.

The video comes out in chunks: each chunk lasts 0.4 seconds and consumes 4 future action vectors, so a default rollout covers 72 frames at 10 Hz — 7.2 seconds of surgery, at 832x480. The commands arrive before the video: either you take them from a recorded procedure or you generate them with a policy model. The model does not decide what to do: it only predicts how the scene evolves.

The real-time numbers

The table comes straight from the model card on Hugging Face (source):

GPUResolutionDenoising stepsFPS
RTX PRO 6000 Blackwell (Workstation)832x480430.0
RTX PRO 6000 Blackwell (Workstation)832x480243.2
RTX PRO 6000 Blackwell (Max-Q)832x480420.2
RTX PRO 6000 Blackwell (Max-Q)832x480231.2
DGX Spark512x288216.9

The operating reference is 10 Hz: that is the rate at which the robot emits commands, so keeping the interactive loop in real time needs at least 10 FPS. The 80 GB card arrives with 3-4x headroom; the 64 GB one with 2x; the DGX Spark, a desktop with 128 GB of unified memory, makes it too, but at reduced resolution.

VRAM: 15.17B parameters exported in FP32 weigh about 60 GB, plus KV cache and VAE. You need an 80 GB GPU, and NVIDIA has not qualified any quantization or lower precision: there is no official way to fit it on a 24 GB consumer card yet. The runtime is FlashDreams (static-shape KV cache, torch.compile, CUDA Graphs, lightweight video decoder) on Linux, with an interactive keyboard or Meta Quest headset stack.

Why it is a concrete use case, not a demo

This is the third model from the Cosmos3 family to arrive on-premise, after the driving simulator Cosmos3-Nano-Sim-Auto and the edge robotics policy Cosmos 3 Edge. The difference in role is the point: it is not the model that makes the difference, it is what you can do with it before touching the physical system.

1. Evaluate surgical policies. The main stated use is evaluating policy models before deployment: you run the policy in the simulation, compare the generated video against what you expect, and catch logic, sequencing, or edge-case errors without involving a physical system. A thousand GPU runs cost energy, not risk.

2. Interactive exploration. The model is also built for teleoperated exploration: you move the robot (or its commands) inside the simulation and see the consequence instantly. For people developing surgical interfaces and workflows, it is a test environment that answers in real time instead of requiring an operating theatre.

3. Synthetic data. The third use is generating synthetic data for robotic cholecystectomy: you augment the 64 real minutes with plausible sequences, a known trick when clinical data is scarce and expensive.

4. All on one GPU, no network. The whole loop runs on a single workstation GPU. No cloud calls, no surgical video crossing a network: the data stays on your hardware. That is the angle that matters when you work with sensitive data and strict policies.

And the limits?

The model is specialized to one embodiment (CMR Versius, domain 25 of the initialization's 32) and one procedure: it does not simulate other robots or other operations. It requires NVIDIA Blackwell or Hopper GPUs with at least 80 GB of VRAM, and quantization has not been qualified: the entry price is a workstation, not a desktop. It generates realistic video, but it is a simulation: it does not replace the surgery, and clinical validation remains mandatory for any medical use.

Where to go next

To understand the family it comes from, read the piece on Cosmos3-Nano-Sim-Auto and the note on the DROID edge robotics policy. To read specifications like "15B parameters", "MoT", or "4 denoising steps" correctly, the guide on how to read a model's specifications remains the reference.

In summary

Cosmos3-H-Dreams shows that local world models have stopped being research demos: on a single workstation GPU you can simulate a robotic surgery in real time, to evaluate policies, explore, and generate synthetic data for robotics. The numbers are NVIDIA's (model card, repository); the constraint is your hardware, not the network.

Article generated with AI

In these reading paths

Share:

Enjoyed this article?

Subscribe to the newsletter for new tutorials and guides.

After signing up you receive a confirmation email (double opt-in). Without confirmation we send nothing. No spam, we promise.

Related Articles

Related resources

Tools and resources from the iaLocale archive relevant to this content.

🚗

NVIDIA Cosmos3-Nano-Sim-Auto

Models

Causal-distilled variant of the Cosmos3 family for real-time driving simulation: generates synchronized multi-camera video (up to 11 standard and fisheye views) and LiDAR, conditioned on a text prompt and WSM controls. 16B parameters, Mixture-of-Transformers architecture, OpenMDW 1.1 license, ready for commercial and non-commercial use.

🤖

NVIDIA Isaac GR00T N1.7 (3B)

Models

Open foundation model for generalized humanoid robot reasoning and skills: multimodal input (language, images, proprioception) and a flow-matching action transformer that conditions an action chunk on vision, language and body state. The N1.7 family comprises 4 post-trained variants (SimplerEnv Bridge, SimplerEnv Fractal, DROID, LIBERO). 3B version, ready for commercial and non-commercial use, open NVIDIA license.

🤖

NVIDIA Cosmos3-Edge-Policy-DROID

Models

Post-trained 4B robot-policy checkpoint on the Cosmos3-Edge base: from natural-language instructions and visual observations of the DROID robot it generates action trajectories for manipulation tasks. Mixture-of-Transformers architecture (autoregressive tower + diffusion transformer), OpenMDW 1.1 license, commercial and non-commercial use.

🩻

NVIDIA Cosmos3-H-Dreams

Models

Real-time, action-conditioned surgical world model for the CMR Surgical Versius system: autoregressively generates the endoscopic video of a robotic cholecystectomy from a frame and the two arms' commands. 15.17B MoT (Qwen3-VL-8B tower + latent-diffusion expert), causal 4-step student with KV cache. 30 FPS at 832x480 on RTX PRO 6000 Blackwell (40GB), 20.2 FPS on Max-Q (80GB) and 16.9 FPS at 512x288 on DGX Spark. NVIDIA Open Model license, released 2026-10-20.