Cosmos 3 Edge: the DROID policy for robotics that runs on the edge

There is a family of models that is not built to answer your questions, but to make a robotic arm move: it is called a policy (a control policy), and the most interesting one right now is Cosmos3-Edge-Policy-DROID from NVIDIA. The name is long, but the concept is simple: a 4-billion-parameter model that looks at where an object is, reads an instruction ("put the banana in the bowl"), and generates a sequence of movements to send to the robot.

The part that concerns us — and that this article tries to measure — is different: how much hardware do you really need to run it on your own hardware, in real time, without sending anything off to the cloud?

What it is, and why "Edge"

Cosmos3 is NVIDIA's platform of omnimodal world models: models that generate video, images, audio and — this is the point — action commands, from combinations of text, images, video, or action trajectories. The architecture is called Mixture-of-Transformers (MoT): two Transformer towers inside the same model, one autoregressive for discrete tokens (text) and one diffusion-based for continuous modalities (image, video, action). That is why a single network can both "talk" and "act".

The Cosmos3-Edge-Policy-DROID checkpoint is the robotics post-trained variant of the base Cosmos3-Edge model. Trained on the DROID robot platform, the input is a combination of language instructions and visual observations, and the output is an 8-dimensional action trajectory (the movement "signature" of that robot).

Why "Edge"? Because the stated goal is deployment on compact hardware — Jetson, DGX Spark — where the model runs next to the sensor, and latency matters more than raw capability.

How big it is, and what is inside

The numbers that matter, all from the official model card:

Characteristic	Value
Trainable parameters	4B
Architecture	Mixture-of-Transformers (MoT)
Precision tested	BF16 (only)
Training data	1.3 billion data points, 393 datasets (2024–2026)
Action space	8D (DROID platform)
License	OpenMDW 1.1 (commercial and non-commercial use)
Release date	2026-07-20

The 4B parameters at BF16 (2 bytes per weight) work out to about 8 GB of weights. It is a model that fits — in principle — on a single consumer GPU or on a Jetson with 16 GB of memory. The point that must be said clearly: NVIDIA has tested BF16 only. The quantized versions (FP4, FP8, FP16) are not officially supported. That changes the math for anyone with less than 16 GB, and we come back to it below.

The model was trained on a huge corpus: 1.3 billion data points aggregated from 393 datasets, including 767 million images, 348 million video clips, and action data. For the robotics part specifically, the checkpoint builds on the DROID dataset, an "in-the-wild" manipulation corpus: 76,000 demonstration trajectories, 350 hours of interaction, collected across 564 scenes, 86 tasks, and 52 different buildings. It is a corpus that covers the usual variety of robotics datasets — objects, lighting, viewpoints — and for that reason the policy generalizes beyond the scenes it was tested in.

How much hardware you need: the table

Here is the heart of the article. The model card publishes end-to-end (E2E) latencies for a single action chunk across a range of platforms, from a 192 GB workstation down to a 16 GB Jetson. The reference test: a 16-step action chunk, one input image, num_frames=17, fps=5.

Platform	Memory	E2E (vLLM)	E2E (PyTorch)
NVIDIA B200 SXM	192 GB	0.99 s	—
NVIDIA H100 SXM	80 GB	1.41 s	1.25 s
NVIDIA RTX PRO 6000 Blackwell	96 GB	1.87 s	1.65 s
NVIDIA DGX Spark	128 GB (unified)	7.66 s	6.56 s
NVIDIA Jetson AGX Thor T5000	128 GB	— (see below)	—
NVIDIA Jetson T2000	16 GB	—	—

The datacenter rows (B200, H100) are there as a reference: when the model has to run "fast" and it does not matter that it is on your desktop. But the rows that matter for the local/edge angle are the last ones.

Real time on Jetson: the real test

For a robot, "real time" has a precise meaning: you must produce an action chunk before the control window expires. The model card measures a 32-step chunk at a 5 Hz control rate, where the budget is 6.4 seconds per chunk. The number to watch is the RTF (Real-Time Factor): available time divided by measured latency. If it is ≥ 1, the model makes it in time.

Jetson	Memory	E2E (vLLM)	RTF at 5 Hz	Real-time?
AGX Thor T5000	128 GB	2.59 s	2.47	Yes
AGX Thor T4000	64 GB	3.24 s	1.98	Yes
Thor T3000	32 GB	4.05 s	1.58	Yes
Thor T2000	16 GB	7.63 s	0.84	No

Read the table this way: the T5000 has 2.47 times of margin against the budget — it runs with room to spare, space for the unexpected. The T2000 (16 GB) drops to 0.84, below the threshold: the model is there, the weights fit, but 5 Hz control does not make it in time. That is the exact boundary between "runs locally" and "runs, but not in real time".

If you do not have a Jetson, the closest reference to a desktop setup is the DGX Spark (128 GB of unified memory): 6.56 s with PyTorch, 7.66 s with vLLM. It is not "real time" for a robot (too slow for 5 Hz), but it is a concrete reference point for anyone who wants to run the policy on a compact workstation without a datacenter GPU.

How to run it

The official setup is a policy server that streams actions to a client that drives the robot (physical or simulated). The flow is: clone cosmos-framework, a Docker image, a container with the NVIDIA runtime, and inside, start the policy server on the checkpoint:

git clone https://github.com/nvidia/cosmos-framework.git
cd cosmos-framework
# build the Docker image (INSTALL_APEX=0)
docker build --build-arg INSTALL_APEX=0 -t cosmos-framework:latest .

Then, inside the container, the policy server:

python -m cosmos_framework.scripts.action_policy_server_robolab \
  --checkpoint_path nvidia/Cosmos3-Edge-Policy-DROID \
  --port 8000 \
  --format-prompt-as-json True

To try it without a robot, NVIDIA points to RoboLab, a simulation benchmark for "task-generalist" policies: the client loads a task (for example BananaInBowlTask) and shows the simulation in real time. It is the most honest way to see the policy at work without spending on hardware: a --num-envs 10 --headless runs ten environments in parallel.

Two things to know before you start. First, the runtime is PyTorch and the declared GPU support is Ampere, Blackwell, and Hopper — no Apple Silicon, no AMD in the official list. Second, you need a Hugging Face token to download the checkpoint, and the model is under the OpenMDW 1.1 license, which allows both commercial and non-commercial use: it is not an "open-weight" model in the strict sense, but it is freely downloadable and reusable.

And the limits?

Honest ones, because the model card says them clearly, and because an article that hides them is not useful to you:

• No real physics. The model has no internal physics simulator: 3D geometry, object permanence, contact dynamics, and the laws of physics are only approximated. An object can appear to disappear or morph, a collision to be unrealistic. Real robotic control needs external validation, additional constraints, and system-level guardrails.
• BF16 only. If your target is a GPU with 8 GB, the quantized version that would let you fit in 8 GB is not officially supported. The real boundary for the desktop is closer to 16 GB than to 8.
• DROID platform only. The 8-dimensional action space is specific to that robot. The model is not a "universal robotic brain" that works with any effector — it is a policy for DROID, and a different robot calls for a different checkpoint or a training run.
• Latency that falls off a cliff when you scale down. From 128 GB (RTF 2.47) to 16 GB (RTF 0.84) the slope is not linear: there is a point where it stops making it in time. If your control needs more than 5 Hz, the T2000 is not your target.
• Out-of-distribution. Quality degrades in environments rarely seen in training and in safety-critical edge cases.

The takeaway: it is a model that runs locally when "local" means a Jetson with 64–128 GB or a workstation with unified memory, and not when it means "the 8 GB consumer GPU on my PC". For that, you need the quantization that, for now, is missing.

What it has to do with local AI

The point is not "I have a robot at home". The point is the pattern: a model big enough to be useful, but compact enough to run on the hardware next to the sensor, without the data going to the cloud. Cosmos3-Edge-Policy-DROID is the most extreme use case of that pattern — because robotics tolerates neither latency nor loss of connection — but it is the same pattern as a RAG running on a Mac, a TTS on a laptop, an embedding on a consumer GPU.

The "how much hardware is enough" boundary shifts from model to model, and the table above is exactly what you need to shift it for yourself too: look at the RTF on your hardware, compare it to your control budget, and decide. If you want to understand how to read these numbers for models that already run locally, the comparison of inference engines (Ollama vs llama.cpp) and the role of vLLM are the right starting points; and if you are short on memory, quantization is the first tool that gets you into fewer GB.

Sources

• NVIDIA official model card — Cosmos3-Edge-Policy-DROID — architecture, dataset, license, and all the E2E/RTF latencies in the tables.
• DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset — 76,000 trajectories, 350 hours, 564 scenes, 86 tasks, 52 buildings.
• DROID Dataset — official site — collection figures and corpus coverage.
• NVIDIA Cosmos Framework — policy server setup and the RoboLab client.

In summary

Cosmos3-Edge-Policy-DROID is a 4B-parameter robotic policy, Mixture-of-Transformers, trained on 1.3 billion data points and the DROID dataset (76,000 trajectories, 350 hours), running in BF16 on about 8 GB of weights. On Jetson AGX Thor T5000/T4000/T3000 it makes real time at 5 Hz (RTF 1.58–2.47); on the 16 GB T2000 it does not (RTF 0.84). On a DGX Spark with unified memory, 6.56 s of latency with PyTorch. It is the most concrete reference that exists today for the boundary between "local AI" and "local AI in real time" — with the honest limit that quantization is not yet supported, and that the model is tied to the DROID platform.

Article generated with AI