Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Embodied AI 101

Embodied AI 101 artwork

Praxis-1: Can Video Pretraining Power Real-Robot Manipulation?

Nov 13, 202613 minAudio
ShareXLinkedInFacebookWhatsAppEmail

Can a model pretrained to generate video become a useful real-robot policy? This episode explains Praxis-1's architecture, measured manipulation performance, and significance for robot foundation models. We cover three findings: how a mixture-of-experts video backbone supplies features to a separate action expert; where Praxis-1 leads the world-action-model category on RoboLab-120; and how its open weights and fine-tuning interfaces support practical experimentation. The discussion distinguishes simulated tabletop evaluation from physical Franka FR3 testing, while explaining action chunking, video-and-action training, and embodiment-specific adaptation. We also connect the architecture to DeVA's decoupled video-action approach and examine what a working roboticist can learn from the released implementation. The central limitation is causal attribution: the final evaluations do not isolate general-video pretraining's contribution to control performance. For researchers working on imitation learning, world models, or language-conditioned manipulation, the episode offers a technical account of what to test next. What changes when your policy's control representation includes an imagined future?

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show.

Praxis-1: Can Video Pretraining Power Real-Robot Manipulation?.

Praxis-1: From Video Pretraining to Robot Control • Runway • Runway Research • Technical report • 2026.

Cold Open.

Picture a robot asked to place a mug upside down on a tray. The destination is easy to describe. Getting there requires a sequence: approach the right object, secure it, rotate it, and make controlled contact with the tray. Now imagine that much of the visual knowledge behind that sequence came from video, before the model learned the commands for this particular arm.

That is Runway’s wager with Praxis-1: general video provides useful knowledge about motion, objects, and task progression, giving robot learning a foundation beyond teleoperated demonstrations. The research question is how to turn that knowledge into executable control.

Runway’s public weights announcement is dated November 11, 2026, following its September introduction. The release puts model weights, inference code, and fine-tuning code under Apache 2.0.

For this episode of Embodied AI 101, we are following the transfer from video generator to robot policy: what changes inside the model, what reaches the robot, and what the experiments establish.

The Idea.

Praxis-1 is a roughly 27-billion-parameter mixture-of-experts video-action policy. Most of its capacity sits in the video backbone; a much smaller action expert produces robot commands. A frozen Wan2.2 video autoencoder converts camera footage into latent representations, while a frozen Qwen3.5-2B text encoder represents the instruction. These components give the policy a compressed visual workspace and a language-conditioning interface.

The action expert is a separate transformer that reads the video backbone’s internal features through cross-attention. It operates on representations inside the generator rather than waiting for a finished movie and analyzing its pixels. This is the central architectural connection: features used to predict the scene’s evolution also inform the motion commands. The implementation keeps video prediction and action prediction as distinct, connected modules.

On the DROID configuration, the model receives an instruction, exterior and wrist-camera observations, and the robot’s pose. It returns an action chunk containing 24 commands at 15 hertz—1.6 seconds of motion—alongside predicted video. The standard execution loop runs the chunk, collects the resulting observations, and calls the policy again. Visual feedback therefore enters at chunk boundaries.

The temporal structure is important. Praxis uses chunk-causal attention: the future chunk reads observed history and its own evolving, noisy representation. During training, recorded history supplies the clean context. Video and actions are learned through rectified flow, which trains the network to turn noise into structured outputs. At inference, both streams advance together through two denoising steps. As the imagined future takes shape, the action stream is refined against that representation.

Training has four stages: general-video pretraining; embodied-video mid-training, dominated by egocentric human footage; joint video-and-action mid-training; and embodiment-specific fine-tuning. Video prediction stays in the objective throughout. The DROID version uses about 277 hours of successful demonstrations for its final adaptation.

The released base checkpoint already contains the shared video backbone and action expert, trained across robot and human footage from multiple embodiments. Adaptation supplies robot-specific input and output projections for proprioception and actions. That creates a useful division of labor: reuse the visual and motion representations, then fit the interface to the target body and its command space.

The documented training paths reflect that division. Full fine-tuning starts from the base and adds the target rig’s heads. For task adaptation on an existing DROID setup, the release provides a LoRA recipe targeting the action expert. These are concrete procedures for converting the released representations into a policy for a particular experimental setup.

There is a clear research lineage here. DeVA, published in July 2026, also separates video and action experts and transfers features from multiple video layers into the action branch. It additionally uses depth and affordance guidance. Praxis belongs to this broader effort to make predictive visual representations useful for control. The distinctive package is Runway’s large video-pretraining foundation, a chunked robot-policy implementation, and publicly adaptable weights.

The Result.

The headline number is 46.2% success on RoboLab-120. NVIDIA’s public leaderboard records 1,200 evaluation episodes, with ten trials per task. As checked on November 13, 2026, Praxis-1 leads the world-action-model category, ahead of FLUX 3 Action, and scores above listed open-weight VLAs such as π0.5. That is the relevant comparison class for the release’s performance claim.

RoboLab is a simulation benchmark for task-generalist policies. Its tasks probe visual recognition, relationships between objects, and procedural requirements such as reorientation. Think selecting objects by appearance, respecting spatial or numerical relationships, and manipulating them into a specified configuration. The suite emphasizes multi-step tasks more strongly than the DROID training distribution and introduces objects beyond its training vocabulary.

That structure makes the benchmark useful for this research question. A policy needs to connect an instruction to a sequence of scene changes. Recognizing a bowl and a block is one component; deciding which object moves, where it belongs, and what constitutes completion adds relational and procedural demands. RoboLab explicitly separates these competencies, giving the aggregate success score a more interpretable task context.

The breakdown also adds useful color. Relative to FLUX 3 Action, Praxis’s gains are concentrated in the moderate and complex task categories. My takeaway from that distribution is to study its future-conditioned representations on compositional manipulation problems. The interesting question for practitioners is which additional task structures become tractable when the policy predicts how the scene should evolve.

The hardware result uses that DROID checkpoint on a Franka FR3 operated by Positronic. Across ten single-item tasks and one hundred trials, Runway reports 88% success, ahead of FLUX 3 Action on the same rig and tasks. The tasks include object placement and drawer closing.

I would use these evaluations for different engineering decisions. The simulation suite helps select task families for a broader comparison. The physical experiment makes the checkpoint worth bringing into a hardware evaluation. Together, they suggest a productive next experiment for a lab: choose manipulation tasks that combine object selection, spatial relationships, and physical execution, then compare the complete control loop. That is where this architecture’s design choices become observable behavior.

The Biggest Weakness.

The biggest weakness is causal attribution: the final-policy evaluations do not isolate how much of the performance comes from general-video pretraining.

Runway’s September announcement does provide a relevant comparison. It reports essentially matching final placement error after fine-tuning when pretraining uses web video versus teleoperated robot video. That supports ordinary footage as useful learning material. The reported comparison uses a placement-error metric, rather than a matched closed-loop task-success ablation of the released policy.

The November compute-scaling study trains longer on the same mid-training mixture before matched fine-tuning. That tests additional optimization of the combined system.

These experiments address related but different questions. Comparing visual data sources asks what footage can teach. Extending mixed mid-training asks whether the assembled learning pipeline keeps improving. The stronger proposition—how much general video substitutes for action-labeled robot experience—needs an experiment that directly measures that substitution.

This matters because the practical decision is about what to collect and what to train next. A team choosing between more general footage and more robot demonstrations needs a data-efficiency curve for completed tasks. An aggregate leaderboard score identifies a capable system; it does not assign credit among the ingredients that produced it.

The decisive experiment would hold architecture, downstream robot data, and fine-tuning budget fixed while changing the pretraining source. It would compare general-video and robot-video initialization under matched pretraining compute, then sweep the amount of action-labeled data. Each variant would face the same held-out tasks in closed-loop execution.

The useful outcome would be concrete: at a specified success rate, how much robot data does general-video pretraining save? That would connect the model’s performance directly to Runway’s proposed change in the economics of robot learning.

Verdict.

Praxis-1 should interest researchers building robot foundation models and labs studying transfer across embodiments. The released base supplies both the video backbone and shared action expert before robot-specific post-training. That makes it a starting point for experiments on how a common representation becomes a policy for a particular body. You can make adaptation itself the research object.

For manipulation teams already using DROID-style hardware, the release offers an especially direct entry point. The model card specifies exterior and wrist cameras, pose-relative Cartesian actions, and reference inference on an 80-gigabyte GPU. Those details define a concrete experimental configuration that can be integrated with an existing robot-learning workflow.

The fine-tuning documentation is useful for planning that work. It separates full adaptation from the base checkpoint and same-rig task adaptation through LoRA. The latter has a documented single-GPU path. It also specifies how observations, commanded poses, normalization, and robot-specific heads enter training. This gives practitioners an implementation to inspect and modify, with an explicit relationship between their dataset and the resulting controller.

Systems researchers should also read Reactor’s deployment write-up. Its serving work exploits the causal structure to reuse clean-context computation and remove work that cannot influence the returned prediction. The important insight is architectural: the attention mask also tells an inference implementation which computations are reusable or unnecessary. Model design and serving design can reinforce each other.

My take is that Praxis makes the video-to-control interface a more accessible research target. Alongside architectures such as DeVA, it encourages a concrete question: which predictive features should an action model read, and how should that interface be trained? That is a productive level of comparison between robot foundation models.

The one-line takeaway: Praxis-1 makes a large video-pretrained model an inspectable, adaptable backbone for robot control.

Outro.

Watch what other labs build from the released base, especially systems that connect future-conditioned control to higher-level task selection. That is a natural next experiment for this architecture.

Return to the mug and the tray. The useful representation must connect a desired scene change with motion the robot can execute. Praxis puts those two predictions inside the same policy call.

For embodied AI, that connection is worth studying closely. Video records how the world changes. A robot policy causes change. The research opportunity is to make knowledge learned from the first increasingly useful for the second.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Can One Human Video Teach Vision-Based Dexterous Manipulation?

    Nov 20, 202613 minAudio

    Can a single human video provide enough guidance for a dexterous robot to learn a useful manipulation skill? This episode explains the proposed division of labor between motion priors, residual reinforcement learning,...

    Credit the people on this episode

  2. ROAM: Can Vision-Language Risk Maps Improve Legged Navigation?

    Nov 20, 202613 minAudio

    How can a legged robot use a vision-language model to navigate trails without asking it to place every foot? This episode explains how ROAM connects semantic scene interpretation to reinforcement-learned navigation and...

    Credit the people on this episode

  3. Doppel: Can Rendered Poses Improve Cross-Embodiment Robot World Models?

    Nov 19, 202613 minAudio

    Can robot world models predict interactions better if we show them how the robot will move? This episode examines Doppel's rendered-pose conditioning, its experimental evidence, and what cross-embodiment prediction...

    Credit the people on this episode

  4. Elastic-WAM: Can Selective Imagination Speed Up Robot Manipulation?

    Nov 19, 202613 minAudio

    Can a robot spend less time imagining the future and still manipulate objects more successfully? This episode explains how Elastic-WAM turns future refinement into a learned computation-allocation problem. We examine...

    Credit the people on this episode

  5. ScaleVLA: Can a Frozen 552B VLM Improve Robot Manipulation?

    Nov 18, 202613 minAudio

    Can a frozen 552-billion-parameter vision-language model make robot manipulation more data-efficient? This episode explains what ScaleVLA connects, what its experiments measure, and what its results establish about...

    Credit the people on this episode