Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Embodied AI 101

Embodied AI 101 artwork

ROAM: Can Vision-Language Risk Maps Improve Legged Navigation?

Nov 20, 202613 minAudio
ShareXLinkedInFacebookWhatsAppEmail

How can a legged robot use a vision-language model to navigate trails without asking it to place every foot? This episode explains how ROAM connects semantic scene interpretation to reinforcement-learned navigation and locomotion through spatial maps. We examine the system's real alpine deployments and controlled experiments involving terrain preferences, warning signs, and foot placement. ([roam-hiking.github.io](https://roam-hiking.github.io/)) Three questions organize the discussion: why distinguish risk from preference and steppability; what does simulation teach the controllers; and which experiments demonstrate that semantic information actually changes behavior? We also place the architecture alongside BehAV's behavioral cost maps and NaVILA's intermediate movement commands, showing how different interfaces divide responsibility between reasoning and execution. The practical focus is the perception-to-control boundary: what information crosses it, how a controller uses it, and what a roboticist can build from the released front end. The central limitation is grounding: correct hazard labels can be lost during segmentation. ([arxiv.org](https://arxiv.org/html/2611.25215v1)) Listen for a technical assessment of what the mountain footage demonstrates - and why choosing the right interface may matter as much as choosing a more capable model.

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show.

ROAM: Can Vision-Language Risk Maps Improve Legged Navigation?.

Bridging vision-language reasoning and learned control for legged navigation in the wild • Takahiro Miki et al. • ETH Zurich and Apple • Science Robotics • 2026.

Cold Open.

The path is flat, the robot is stable, and the destination is straight ahead. Then a construction sign tells it to leave the trail. Nothing about the ground’s shape has changed. The correct behavior has.

In ROAM’s campus experiment, the robot takes the grass around the closed section, then returns when another sign reopens the path. Language changes the route without anyone teaching a new gait.

That is a different embodied-AI question from making a robot walk over rough terrain. How do you connect a model that understands a warning to a controller that has to execute the next step?.

Should the model produce a direction? Draw a trajectory? Or describe which surfaces deserve caution and let the rest of the system decide where to move?.

ROAM explores that last option, connecting terrain interpretation to both navigation and individual foot placements.

The Idea.

ROAM stands for Robust Open-world Autonomous Mobility. Its organizing decision is to make the vision-language model a producer of map annotations. The original deployments used GPT-4V to inspect camera images and return terrain labels with control-relevant values. Recent textual summaries provide context across successive observations.

An operator selects coarse waypoints on a satellite map. A global planner supplies a reference route, which becomes nearby goals for local navigation.

The semantic output separates three questions that roboticists often collapse into traversability. Risk asks how undesirable a region is. Steppability asks whether a foot may land there. Preference asks which otherwise usable region the robot should favor. Ordinary grass can be physically usable while the trail remains preferable. This distinction gives the planner room to express a detour without treating every nonpreferred surface as an impassable obstacle. The prompt also describes the robot’s capabilities and explicitly asks it to update terrain judgments when signs change the context.

Next comes SAN, the Side Adapter Network for open-vocabulary segmentation. SAN adapts frozen CLIP features to recognize image regions using supplied category names. Here, it locates the categories selected by the VLM. ROAM converts those detections into spatial value images and fuses them into an elevation map. The resulting representation carries terrain geometry alongside semantic information.

Two learned controllers consume that representation. A local navigator generates body-velocity commands toward its goal; a locomotion policy executes those commands. This follows the control lineage of Joonho Lee and colleagues’ 2024 wheeled-legged navigation system, where navigation is trained together with an executable locomotion skill. The upper-level policy learns to work through the movement capabilities of the lower-level policy.

ROAM adds semantic guidance at both levels. The locomotion policy receives steppable information for foothold selection. Navigation preferences also modify the navigator’s terrain input: areas adjacent to a preferred path can receive a virtual height increase, encouraging the robot to remain on the path.

The control training uses simulation and abstract terrain observations. For locomotion, procedurally generated no-step patches become extra inputs, and foot contact with those patches incurs a penalty. Navigation similarly learns to avoid risky regions. The training worlds therefore need varied geometry and semantic fields, rather than photorealistic replicas of every possible hiking environment.

This fits a well-established training pattern: a teacher learns with privileged simulation information, then a deployable student learns from observations available to the robot. Lee’s preceding system uses that separation to learn robust movement before training the navigation layer.

The important conceptual move is separating the identity of a hazard from the response to it. A controller can learn how to negotiate a forbidden patch before a perception system decides which real-world object should instantiate that patch. That is an attractive division of the learning problem: acquire visual concepts from broad pretraining, and acquire physical responses through simulated interaction.

Finally, the clocks are decoupled. Segmentation continues processing fresh images using the latest VLM labels. Mapping and control keep updating between language-model queries. A separate sign detector can pause the robot and request a fresh interpretation. Scene understanding supplies context; the faster components supply spatial updates and movement.

The Result.

The headline is 96.7% time-based autonomy over a 10.1-kilometer Zermatt hike on ANYmal D. The metric measures the share of mission time spent operating autonomously.

The route crossed gravel, rock, grass, lakeside terrain, and railway-adjacent sections. A second deployment around Seealpsee included narrow waterside paths, cows, and forest. These were real alpine deployments of the integrated system.

The value of that distance is integration time. Perception, localization, mapping, navigation, and locomotion must remain coordinated as the environment changes. A long route exposes interactions among modules that a single obstacle crossing might never exercise. For a systems paper, sustained operation is itself an important result.

On the controlled campus course, ROAM faced the geometry-only Field Local Planner and image-goal navigator ViNT. ROAM followed the sign-directed detour and avoided the mannequin; the baselines entered the closed route or contacted the mannequin.

The sign experiment is particularly revealing because the interpretation changes while the physical surface remains available. The grass does not suddenly become easier to walk on. Its role in the task changes. That makes the demonstration a test of contextual decision-making, rather than simply another terrain-recognition example.

The local-navigation ablation supplies a more direct intervention on the system: disabling semantic guidance changes the route while retaining the learned navigator. With semantics, the robot favors the path; without them, it crosses grass and railway areas. That connects the behavioral difference to the added information channel.

Staged foothold experiments push the same idea down to contact selection. Warning signs make pallets or cans off-limits, and the robot changes its foot placements accordingly. Meaning affects both the body’s route and the surfaces its feet contact.

A separate image-space evaluation compares output abstractions. For that test, the authors replace ROAM’s learned navigator with A* over its semantic map. It produces safer, more goal-directed paths than prompting the VLM to emit coordinates or select marked points.

My take from these experiments is that the most useful evidence is behavioral. The question is whether semantic interpretation changes a downstream decision in the intended way. Changing a route, reversing a terrain preference, and moving a foothold are observable consequences. Together, they make a stronger case for the interface than a convincing verbal explanation of the scene would.

The Biggest Weakness.

The weakest link is grounding: a correct hazard label can still become an incorrect spatial map.

At Zermatt, the VLM identified a fence as dangerous, but segmentation failed to locate it accurately and the robot collided. The supplementary failure analysis identifies segmentation as the dominant cause of hiking interventions.

That failure goes directly to the paper’s central architectural claim. ROAM assigns semantic judgment and spatial localization to different components. SAN recognizes categories through predicted image regions; naming a category does not ensure that its physical extent will be recovered correctly. The resulting engineering consequence is straightforward: if a hazard’s pixels receive the wrong classification, the semantic warning can disappear before reaching the controller.

The distinction is between obeying a map and validating a map. A controller can be excellent at avoiding marked hazards while remaining vulnerable to real hazards that its input never marks. The reasoning model can produce an entirely sensible answer, and the downstream control policy can respond consistently to its observations, yet their composition can still produce the wrong action.

This also changes how to read time-based autonomy. Long, uneventful stretches contribute heavily to that metric. Brief encounters with a fence or an ambiguous terrain boundary can determine whether the mission ends safely. High autonomous operating time and dependence on a human at decisive moments are therefore compatible.

The decisive next experiment would repeatedly present hazards that are easy to name and difficult to segment, then trace each through label, mask, map, and action. I would want the system to stop or request assistance when it cannot locate a hazard it has already recognized.

Measure successful handling of those encounters, including appropriate abstention. That directly tests whether the proposed bridge preserves the safety-relevant meaning that motivated building it.

Verdict.

ROAM should interest teams that already have useful legged locomotion and want richer decisions about where their robots should travel. The reusable idea is the interface between scene interpretation and control.

Place it beside BehAV, the University of Maryland system that converts human behavioral instructions into a cost map and combines that representation with lidar for model-predictive outdoor navigation. BehAV establishes a closely related design pattern: behavioral meaning can enter navigation through spatial costs. Its instructions explicitly specify behaviors such as following particular surfaces or avoiding particular objects.

NaVILA provides another useful comparison. It fine-tunes a vision-language model to produce intermediate movement instructions, which a visual reinforcement-learning policy executes. Its training combines navigation data, human video, and question-answering tasks. The high-level interface expresses the next movement, while the locomotion layer handles execution.

These approaches suggest a productive way to think about embodied model design. Ask what information should cross the boundary between reasoning and action. A movement instruction, a waypoint, and a semantic field allocate different responsibilities to perception, planning, and control. They also imply different training data.

For practitioners, that makes the interface decision worth making before the model decision. Do you want a language model to propose the next movement, or to modify the world representation that your navigation stack already consumes?.

The released ROAM front end makes the latter option concrete. It includes VLM prompting, SAN segmentation, semantic-image generation, ROS integration, and an image-space planning demonstration. Those are useful entry points for experimenting with the perception-to-map boundary.

The strongest contribution is this tested division of labor: use language to interpret the operational meaning of terrain, then use an explicitly spatial representation to connect that interpretation to learned movement.

The takeaway: transfer world knowledge through semantic maps, and teach the physical response through interaction.

Outro.

The next result to watch is reuse across robot bodies: keep the semantic interface stable while changing the locomotion skill underneath it. That would be a concrete experiment in which parts of navigation knowledge transfer.

The image to remember is a warning changing a map, a map changing a route, and a trained controller carrying that decision through to the feet. Follow those interfaces and embodied intelligence becomes something you can inspect, intervene on, and test.

For a working roboticist, that is where this paper gets interesting.

This is Embodied AI 101.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Can One Human Video Teach Vision-Based Dexterous Manipulation?

    Nov 20, 202613 minAudio

    Can a single human video provide enough guidance for a dexterous robot to learn a useful manipulation skill? This episode explains the proposed division of labor between motion priors, residual reinforcement learning,...

    Credit the people on this episode

  2. Doppel: Can Rendered Poses Improve Cross-Embodiment Robot World Models?

    Nov 19, 202613 minAudio

    Can robot world models predict interactions better if we show them how the robot will move? This episode examines Doppel's rendered-pose conditioning, its experimental evidence, and what cross-embodiment prediction...

    Credit the people on this episode

  3. Elastic-WAM: Can Selective Imagination Speed Up Robot Manipulation?

    Nov 19, 202613 minAudio

    Can a robot spend less time imagining the future and still manipulate objects more successfully? This episode explains how Elastic-WAM turns future refinement into a learned computation-allocation problem. We examine...

    Credit the people on this episode

  4. ScaleVLA: Can a Frozen 552B VLM Improve Robot Manipulation?

    Nov 18, 202613 minAudio

    Can a frozen 552-billion-parameter vision-language model make robot manipulation more data-efficient? This episode explains what ScaleVLA connects, what its experiments measure, and what its results establish about...

    Credit the people on this episode

  5. VLA-0: Can Integer Action Decoding Simplify Robot Manipulation?

    Nov 18, 202613 minAudio

    Can ordinary numerical text turn a vision-language model into a capable robot policy? This episode explains how VLA-0 approaches robot manipulation through integer action decoding while retaining the base VLM's...

    Credit the people on this episode