Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Embodied AI 101

Embodied AI 101 artwork

SLA: Does 3D Pretraining Improve pi0.5 Robot Manipulation?

Nov 9, 202613 minAudio
ShareXLinkedInFacebookWhatsAppEmail

Can a robot learn better 3D manipulation without running an additional perception model at deployment? This episode examines how SLA extends π0.5 through spatially supervised continual pretraining and downstream adaptation. We cover three questions: how a frozen OmniVGGT teacher shapes intermediate visual features; why adaptive layer selection and state dropout belong in the training recipe; and what the moving-target experiment measures. The headline result is a rise from 42% to 76% success on a real-world bimanual Piper fruit-manipulation condition requiring grasping, handover, and placement. We connect that experiment to simulation-based robustness evaluation in RoboTwin and explain why applying alignment at both training stages matters. We also place SLA in the lineage of Spatial Forcing, distinguishing feature-alignment machinery from the expanded pretraining strategy. The central limitation is causal attribution on hardware: the reported comparison evaluates the combined recipe rather than a matched control isolating geometric alignment. For roboticists building vision-language-action policies, the episode offers a concrete research question: how much useful physical understanding can training leave inside the policy you already deploy?

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show.

SLA: Does 3D Pretraining Improve pi0.5 Robot Manipulation?.

SLA: Scaling Spatial-Language-Action Model via Continual Pre-training • Fuhao Li, Wenxuan Song et al. • HKUST (Guangzhou), OLA Dimensions, BAAI, Tsinghua University • arXiv • 2026.

Cold Open.

Picture a robot reaching for a piece of fruit. A person moves it just before the gripper arrives. The robot must update its reach, secure the fruit, pass it to its other arm, and finish placing it in a bowl.

The instruction has not changed. The geometry has.

For a manipulation policy, that interruption asks a revealing question: is the next action grounded in the current scene, or in the trajectory that usually works?.

SLA approaches that question through training. The HKUST Guangzhou-led team starts with Physical Intelligence’s π0.5 and adds supervision from a frozen spatial perception model. The deployed policy keeps the geometric teacher outside its inference path.

That creates an interesting engineering proposition. Instead of making the robot run another perception system whenever it moves, teach its existing visual representations to carry more useful spatial information. Where should that learning happen: during adaptation to one task, or earlier, across a broad collection of robot experience?.

The Idea.

The starting point is an existing VLA. In π0.5, a vision-language backbone processes observations and instructions, while a flow-matching action expert produces continuous action chunks. SLA retains that machinery and adds a representation-learning objective alongside action prediction. Its implementation captures intermediate visual features from the language backbone and supervises them through an auxiliary projector.

Think of this as asking the policy to satisfy two requirements simultaneously. It must predict the demonstrated movement. It must also organize its visual features so they resemble features from a geometry-specialized teacher. The supervision targets internal representations rather than requiring the policy to generate a reconstruction as an intermediate control output. The released teacher integration extracts features with its reconstruction heads disabled.

That teacher is OmniVGGT. Its important property is flexibility: it can work from RGB images alone or incorporate available depth and camera calibration. Its training exposes it to different combinations of geometric inputs, allowing it to use richer annotations without requiring every example to contain them. This suits a robot-data mixture assembled from different collection systems.

The alignment is spatially organized. Teacher features are resized to correspond to the policy’s visual-token grid, with positional information retained. The objective is therefore more specific than matching a single global scene descriptor: it supervises features associated with corresponding image regions.

SLA also learns which transformer layers should contribute. A projector maps visual embeddings into the teacher’s feature space, and content-dependent gates combine projected features across layers. This gives the alignment objective access to several representation depths rather than committing to one fixed layer. These gates belong to the training branch, not a new action-selection mechanism.

Another ingredient is state dropout. On selected training samples, the robot-state segment is omitted from the tokenized input while the instruction remains. The released configuration makes this a probabilistic training choice. The intended effect is greater dependence on visual observations: proprioception should help control the robot, rather than simply identify which demonstrated movement comes next.

The training schedule gives these components their broader purpose. SLA applies alignment during continual pretraining and again during downstream adaptation. Its corpus draws from six datasets, including DROID, RoboCOIN, AgiBot World, InternData-A1, MolmoAct, and RoboCasa365. A unified processing pipeline reconciles their action conventions and geometric annotations.

The direct predecessor is Spatial Forcing, which already aligned intermediate VLA representations with a frozen 3D model during fine-tuning. That work established the teacher–student mechanism and evaluated it with OpenVLA and π0. SLA advances the training strategy: develop spatially supervised representations across heterogeneous robot experience, then reinforce them during specialization.

That distinction is the central idea. A task adaptation stage starts from a representation already shaped by geometric supervision. The auxiliary machinery teaches the policy; it does not become another subsystem the deployed controller must consult.

The Result.

The headline concerns moving-fruit manipulation: success rises from 42% for π0.5 to 76% for SLA. On the bimanual Piper platform, each method receives fifty single-object evaluation rollouts. Success requires the complete grasp, handover, and placement sequence. Policies are post-trained on one hundred base-scene demonstrations, without additional moving-target demonstrations.

That protocol asks whether a learned skill survives an intervention. The robot already knows the requested sequence. What changes is the relationship between its gripper and the object while execution is underway. Completing the sequence requires the policy to respond to fresh visual evidence and then continue toward the original goal.

Spatial Forcing finishes between π0.5 and SLA on this condition. That gives the comparison a useful progression: the starting policy, the earlier spatial-alignment approach, and the expanded training recipe. The practical question becomes what happens when spatial supervision moves earlier in the learning process.

Simulation supplies a broader stress test. RoboTwin 2.0 evaluates bimanual manipulation with structured variation in clutter, lighting, backgrounds, tabletop height, and language. Its Clean2Random track adapts policies using clean demonstrations and tests them under randomized conditions. The evaluation therefore asks whether familiar manipulation skills remain usable when their surroundings change.

SLA’s useful ablation independently switches alignment on or off during continual pretraining and post-training. Across the tested data scales, applying it during both stages gives the strongest RoboTwin Clean2Random results.

That experiment addresses a more interesting question than whether additional robot demonstrations help. It asks whether additional experience becomes more useful when the representation is supervised differently. The observed ordering supports treating spatial learning as part of the foundation-building process and preserving it during specialization.

The benchmark selection also matters. LIBERO-Plus systematically perturbs camera viewpoints, robot initial states, object layouts, instructions, and visual conditions. Its design separates robustness across these changes instead of collapsing everything into success under one familiar configuration.

These evaluation designs are complementary. A moving target asks for responsiveness during execution. Randomized environments ask for robustness across episodes. Together, they provide a useful way to investigate whether a representation supports action beyond the exact visual conditions of the demonstrations.

The Biggest Weakness.

The real-world headline does not isolate the contribution of 3D supervision. The hardware table lacks a control matching SLA’s continual pretraining, state dropout, and action processing while disabling only geometric alignment. The paper’s RoboTwin ablation also shows a substantial performance loss when state dropout is removed from the same continually pretrained starting point.

That makes causal attribution the strongest objection.

A moving-fruit task is exactly where encouraging visual dependence could help. A policy that pays more attention to its current images may become more responsive even before we establish which geometric information its features contain. State dropout is therefore a plausible contributor to the improvement, not evidence that the spatial teacher is irrelevant.

The simulation alignment ablation is meaningful evidence against the explanation that the gains come merely from additional data. The remaining question is whether the same separation holds for the physical result attracting attention. Winning with a complete training package establishes which system performs better. It does not assign the improvement to individual ingredients.

The decisive hardware experiment would keep the starting checkpoint, pretraining mixture, training budget, action representation, and state-dropout rule fixed. Train one policy with geometric alignment and one without it. Adapt both on identical demonstrations, then evaluate them under the same moving-target protocol.

Matching downstream demonstration counts alone does not match the models’ learning histories.

For practitioners, this distinction determines what to adopt. Should you invest in a spatial teacher, or does a substantial share of the benefit come from strengthening visual dependence while learning from broader experience? Those possibilities motivate different changes to an existing training pipeline.

The missing comparison is valuable because it would turn a successful combined recipe into a more portable engineering principle: a measured contribution from geometric supervision under otherwise matched physical-task training.

Verdict.

If you maintain a π0.5-based manipulation stack, SLA deserves attention. Its engineering appeal is the decision to improve representations already used by the controller. The intervention happens where the policy learns what information to preserve, rather than through another decision-making module placed between perception and action.

There is a concrete implementation to inspect. The official repository documents loading an SLA pretrained checkpoint, computing normalization statistics, running alignment training, and serving the resulting policy through an OpenPI-based workflow. It also points to benchmark checkpoints and real-world demonstration data. That gives a lab a starting point for experiments beyond reproducing the appearance of a demo.

For foundation-model teams, the useful principle is that training supervision and deployment inputs can differ. Physical Intelligence’s π0.5 work uses heterogeneous robot experience and semantic supervision to develop a reusable policy. OmniVGGT supplies a complementary mechanism for using heterogeneous geometric annotations. Together, these approaches suggest treating every available annotation as a potential learning signal, rather than requiring it to become a permanent sensor dependency.

That is the practical direction I would pursue. A robot does not need to carry every model that helped train it. Spatial teachers can belong to training infrastructure, alongside demonstration curation and augmentation. Their value is in the behavior they leave behind.

For representation researchers, the interesting question is when useful spatial structure should enter the policy. A shared pretrained representation offers a place to learn something once and reuse it across later adaptations. This shifts attention from designing another input pathway to designing better supervision and a better training schedule.

For manipulation engineers, it suggests a focused research bet: inspect what the visual tokens are being asked to encode, not just how much data the action head receives.

Takeaway: make geometry a training objective, then let the existing policy carry it into manipulation.

Outro.

Return to that piece of fruit. The robot has learned a skill; now the world changes halfway through using it. Its next action must respond to the scene in front of it.

SLA puts representation learning at the center of that moment. Watch whether training-only spatial teachers become a reusable ingredient across other policy backbones and robot embodiments.

The appealing destination is straightforward: richer physical understanding in the weights, with a familiar policy at the controls. Better manipulation can begin with changing what the model learns to see.

This is Embodied AI 101.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Can One Human Video Teach Vision-Based Dexterous Manipulation?

    Nov 20, 202613 minAudio

    Can a single human video provide enough guidance for a dexterous robot to learn a useful manipulation skill? This episode explains the proposed division of labor between motion priors, residual reinforcement learning,...

    Credit the people on this episode

  2. ROAM: Can Vision-Language Risk Maps Improve Legged Navigation?

    Nov 20, 202613 minAudio

    How can a legged robot use a vision-language model to navigate trails without asking it to place every foot? This episode explains how ROAM connects semantic scene interpretation to reinforcement-learned navigation and...

    Credit the people on this episode

  3. Doppel: Can Rendered Poses Improve Cross-Embodiment Robot World Models?

    Nov 19, 202613 minAudio

    Can robot world models predict interactions better if we show them how the robot will move? This episode examines Doppel's rendered-pose conditioning, its experimental evidence, and what cross-embodiment prediction...

    Credit the people on this episode

  4. Elastic-WAM: Can Selective Imagination Speed Up Robot Manipulation?

    Nov 19, 202613 minAudio

    Can a robot spend less time imagining the future and still manipulate objects more successfully? This episode explains how Elastic-WAM turns future refinement into a learned computation-allocation problem. We examine...

    Credit the people on this episode

  5. ScaleVLA: Can a Frozen 552B VLM Improve Robot Manipulation?

    Nov 18, 202613 minAudio

    Can a frozen 552-billion-parameter vision-language model make robot manipulation more data-efficient? This episode explains what ScaleVLA connects, what its experiments measure, and what its results establish about...

    Credit the people on this episode