Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Embodied AI 101

Embodied AI 101 artwork

HexaShell: Can Debugging Code Generalize RGB Robot Manipulation?

Oct 19, 202612 minAudio

Can a robot improve at manipulation by debugging code instead of updating neural-network weights? This episode explains how HexaShell turns robot experience into reusable tools, and what its RoboDojo score actually establishes. We cover three questions: how calibrated RGB observations become metric positions; why the reported 39.7% success rate uses category-balanced scoring; and how holding out layouts differs from holding out tasks. The experiments use a simulated dual-arm ARX X5 with head and wrist cameras, camera calibration, and proprioception. Along the way, we connect the design to Code as Policies and URAI, examine an inspectable geometric helper, and separate tool development from execution with frozen tools. The central limitation is a mismatch in task-level learning conditions: HexaShell develops tools on tasks in a category that RoboDojo's published training split reserves for transfer. For researchers building language-driven manipulation systems, the episode offers a concrete way to think about code as learned policy state. If software is part of what a robot learns, where should the training boundary be drawn - and what would convincing generalization look like?

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show.

HexaShell: Can Debugging Code Generalize RGB Robot Manipulation?.

HexaShell: A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents • Hongcheng Gao, Alex Zhou et al. • HexaFuture • arXiv preprint • 2026.

Cold Open.

A robot pushes a T-shaped block onto its target, then “corrects” it out of place. The culprit is a camera-area calculation. HexaShell’s optimizer repairs that calculation, adds a regression test, and the next attempt succeeds.

The durable result is a software change that the next episode can reuse. The foundation model stays fixed; the robot’s working toolkit improves. A failed manipulation has become something closer to a failed software test.

We usually ask what a policy has learned about the world. This experiment asks where that learning should live. In weights? In the prompt? Or in a function that measures an object more reliably every time it runs?.

If experience can become executable, the question shifts: what should the robot repair, and how far does the repair travel? Today on Embodied AI 101: robot learning with a code diff.

The Idea.

HexaShell operates a simulated dual-arm ARX X5 in RoboDojo’s Isaac Sim environment. It uses Codex with GPT-6-Astra at medium reasoning effort for both the acting agent and the tool-repair agent. During execution, the actor examines the scene and chooses commands. Between development attempts, the repair agent changes the software available to it.

Those roles share a deliberately small robot-facing interface, the command. Commands move or orient a gripper, change its opening, request observations, or finish an episode. Below that interface, inverse kinematics and timed joint trajectories convert targets into motion. The language model chooses what to do; conventional control infrastructure carries it out.

RGB describes the visual input, not the entire observation. The system also receives camera calibration and the robot’s own state, including joints and gripper poses. It sees one head camera and two wrist cameras. Its tool API excludes simulator object poses and task-scoring code. That is the precise meaning of controlling from observable information here.

Now comes the useful division of labor. The vision-language model selects a target in an image. Geometry turns that selection into a location. A calibrated pixel defines a ray, not a unique point. Intersect that ray with a measured plane, or match the point across two views and triangulate. If the wrist needs to move for a better view, that motion is an action and uses the episode’s movement budget.

Even the table height is measured. The released helper tries candidate heights, projects one camera’s image into another, and chooses the height where their appearance agrees. It retains that measurement within the episode and exposes diagnostics about the agreement between views. This is classical multiview vision packaged as a callable tool: a numerical procedure that supplies metric structure to the agent.

Tools are Python modules paired with short interface descriptions. The actor sees how to call them; the implementation runs behind the API. Each task enables a specific collection, alongside shared utilities.

Development proceeds through attempts, logged failures, code edits, tests, and retries. Tools are developed on ten layouts per task, then frozen for evaluation. The RGB tools start from earlier depth-based versions; the model’s weights stay fixed.

This sits in a clear research lineage. Jacky Liang and colleagues’ Code as Policies established that language models could compose robot API calls, numerical computation, and feedback loops into executable policies. It made code an expressive action representation rather than merely a way to print a plan.

URAI, introduced by overlapping HexaFuture authors in Make Code as Policy Great Again, already separated an agent that builds tools from one that selects and executes them. Validated revisions persisted across episodes. HexaShell brings that division together with a minimal command interface, RGB geometry, and evaluation across the full manipulation suite.

The Result.

The headline is 39.7% success on RoboDojo’s category-balanced metric. VPP2, the strongest learned policy in the paper’s October 14, 2026 leaderboard snapshot, scores 32.3%. Those are the numbers behind the authors’ state-of-the-art claim.

Understand the denominator. RoboDojo divides its 42 simulation tasks into generalization, memory, precision, long-horizon execution, and open-vocabulary tasks. The overall metric gives each category equal weight, with generalization combining standard and cluttered scenes. The headline describes balanced coverage across capabilities, rather than the fraction of all recorded episodes that succeeded.

Evaluation uses ten new layouts per task, one attempt per layout, with frozen tools and no optimizer. RoboDojo’s checker judges success. The prompt contains the instruction, a task recipe, and development notes. The actor still responds to fresh observations; its software stays fixed.

The supporting comparison I find most useful is the depth experiment. On matched development layouts, the geometric RGB version outperforms the depth-network version overall. Pushing the T-shaped object and playing the xylophone improve substantially. This comparison focuses attention on how the system obtains coordinates for the controller.

For a roboticist, that is an actionable result. A learned depth estimator and calibrated multiview geometry solve related problems through different assumptions. Here, exploiting the known camera arrangement and actively obtaining views pays off. The engineering lesson is to invest in explicit measurements alongside a capable model, rather than treating every spatial decision as another language-model prediction.

The task results also show why the abstraction is interesting. HexaShell completes language-conditioned stacking, remembered button-press sequences, and tic-tac-toe layouts. Those episodes require symbolic choices to become concrete manipulations: which object, which order, and which target matters now.

Read the result as evidence for a particular allocation of work. Semantic interpretation remains flexible. Numerical operations become executable and repeatable. The policy is the combination: a model making decisions through a library that has already absorbed experience. That is the architectural story behind the aggregate score, and it connects directly to URAI’s separation of tool construction from feedback-driven execution.

The Biggest Weakness.

The biggest weakness is that the headline is not a matched test of task generalization. HexaShell develops tools for every evaluated task. RoboDojo’s published training split deliberately excludes its Open-category tasks, because that category is intended to test whether policies can recombine existing skills for unseen task specifications. Those are different learning conditions.

Consider language-conditioned block stacking. A policy encountering that task without task-specific training is being tested on transfer. An agent that has already spent development episodes repairing tools for that task is being tested on how well its adapted solution handles another layout. Both experiments are useful. They answer different questions.

Freezing the tools before evaluation creates a meaningful boundary around the new layouts. It does not make the task itself unseen. Moving learned knowledge into code changes its representation, not its status as learned knowledge. Persistent experience is exactly what makes tool evolution compelling. A comparison has to count software adaptation as adaptation.

This matters directly to the score: Open is an equally weighted category and one of HexaShell’s strongest. Its contribution cannot automatically be read as evidence of the same transfer capability that the benchmark’s original training split was designed to measure.

No weight updates is not synonymous with zero-shot evaluation. Here, the optimizer can change mechanisms used to interpret observations and execute actions. Those changes belong on the learning side of the comparison.

The decisive experiment is straightforward: build the tool library on a declared training partition, freeze it, and evaluate unseen task specifications under the same rules as competing policies. Alternatively, give every method a matched task-adaptation phase and label the comparison accordingly. That would distinguish transferable tools from efficient specialization without changing the central method.

Verdict.

Robot-agent builders should care because HexaShell makes the interface itself a place where capability can accumulate. The productive question is not only which model to call. It is which recurring operation should become a tested tool that the model can invoke again. That reframes agent development around reusable computations.

Perception and control researchers should recognize a familiar advantage in the architecture. Object selection and geometric estimation are separated. That separation gives you a concrete place to investigate a bad command: the chosen image feature, the recovered location, or the motion generated from it. Each software component becomes something you can test separately.

The released table-height implementation is a useful artifact to inspect. It exposes measurement diagnostics and can run on saved observations. You can examine how image agreement changes with the hypothesized plane without rerunning an entire manipulation episode. This is the kind of local, inspectable computation that makes an agent’s behavior easier to study.

For robot-learning researchers, the conceptual payoff is the location of memory. Code as Policies made programs a policy representation. Persistent tool revision makes a program library a record of experience. A successful repair changes the computation available on future episodes, not just the explanation attached to the last failure.

My take: HexaShell is a strong demonstration of coding agents as robotics software engineers. Its most useful lesson is to combine flexible semantic decisions with explicit measurement and reusable execution. Treat the library as part of the policy you are building. Give it clear inputs, observable outputs, and a development history that can be inspected. The practical prize is a controller whose experience produces better tools for the next decision.

Outro.

The next development worth watching is a library that grows in useful structure: better measurements, clearer motion interfaces, and reusable recovery procedures. That is a research direction, not simply a larger collection of generated scripts.

Return to the opening: the failed manipulation becomes a regression test and a changed function.

That is the promise to carry forward. Experience can become an executable asset. A robot can carry something from one episode into the next that another engineer can read, inspect, and improve.

Sometimes the most important thing a robot learns is not another response. It is a better function.

This has been Embodied AI 101.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Can One Human Video Teach Vision-Based Dexterous Manipulation?

    Nov 20, 202613 minAudio

    Can a single human video provide enough guidance for a dexterous robot to learn a useful manipulation skill? This episode explains the proposed division of labor between motion priors, residual reinforcement learning,...

    Credit the people on this episode

  2. ROAM: Can Vision-Language Risk Maps Improve Legged Navigation?

    Nov 20, 202613 minAudio

    How can a legged robot use a vision-language model to navigate trails without asking it to place every foot? This episode explains how ROAM connects semantic scene interpretation to reinforcement-learned navigation and...

    Credit the people on this episode

  3. Doppel: Can Rendered Poses Improve Cross-Embodiment Robot World Models?

    Nov 19, 202613 minAudio

    Can robot world models predict interactions better if we show them how the robot will move? This episode examines Doppel's rendered-pose conditioning, its experimental evidence, and what cross-embodiment prediction...

    Credit the people on this episode

  4. Elastic-WAM: Can Selective Imagination Speed Up Robot Manipulation?

    Nov 19, 202613 minAudio

    Can a robot spend less time imagining the future and still manipulate objects more successfully? This episode explains how Elastic-WAM turns future refinement into a learned computation-allocation problem. We examine...

    Credit the people on this episode

  5. ScaleVLA: Can a Frozen 552B VLM Improve Robot Manipulation?

    Nov 18, 202613 minAudio

    Can a frozen 552-billion-parameter vision-language model make robot manipulation more data-efficient? This episode explains what ScaleVLA connects, what its experiments measure, and what its results establish about...

    Credit the people on this episode