Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Embodied AI 101

Embodied AI 101 artwork

Show-Harness: Can Small VLMs Replace Frontier Models in Robot Control?

Oct 23, 202613 minAudio

Can a small vision-language model take over robot control from a frontier agent? This episode explains what Show-Harness actually compresses, how its semantic action interface works, and where its strongest results come from. We cover three central questions: why a compact movement vocabulary changes the learning problem, how GUMI turns browser-based demonstrations into policy-training examples, and what the real-robot evaluation establishes. The method combines supervised action imitation with LoRA adaptation of Qwen3.5-2B, using demonstrations from Franka and AgileX robots alongside a separate ManiSkill and RoboLab simulation corpus. ([huggingface.co](https://huggingface.co/showlab/Show-Harness-VLMs)) We examine the manipulation results, the action-representation ablation, and the relationship to Code as Policies and SmolVLA. The key limitation is that the standalone compact controller does not preserve the frontier system's reasoning-task performance. ([arxiv.org](https://arxiv.org/html/2609.10522v1)) For roboticists working on data-efficient imitation learning and local robot control, the episode offers an architectural takeaway: the interface between model and machine can be as important as the model itself. What becomes easier to learn when the robot's actions become easier to express?

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show.

Show-Harness: Can Small VLMs Replace Frontier Models in Robot Control?.

Show-Harness: Just a VLM Agent Can Play Robots • Yanzhe Chen et al. • Show Lab, National University of Singapore • arXiv • 2026.

Cold Open.

Picture a robot looking at a banana beside a plate. The next useful decision might be embarrassingly simple: move left, move down, close the gripper. What would happen if we made those decisions the model’s entire output language?.

That is the engineering question behind Show-Harness. Its authors give the robot something resembling a game controller: a small vocabulary of movements whose physical meanings are fixed in advance. A frontier vision-language model can use that interface directly; a compact model can learn to use it from demonstrations.

Imagine the practical payoff. You demonstrate through the same buttons the policy will later press. You train an adapter, put the model beside the robot, and keep observing after each movement.

The interesting question is what that interface buys you. Does it simply make outputs easier to parse, or does it change how much robot learning you need? And how much of the larger agent can fit inside that smaller controller?.

The Idea.

The compact controller consumes a scene view, a wrist-camera view, the task instruction, and a short record of recent movements. It returns one action symbol. The released basic policy chooses among six translation directions, closing the gripper, opening it, and declaring completion. Each iteration brings fresh camera observations back into the decision process.

Those symbols go to a robot-specific interpreter. The model selects a movement; the interpreter translates it into the corresponding hardware command. In the released demonstrations, translation units share a two-centimeter step across Franka and AgileX. The dataset also fixes a common directional convention, reconciling the rigs’ different viewpoints.

Think through a hypothetical grasp. The gripper is slightly behind the banana. The policy chooses a forward movement, examines the resulting wrist image, then decides whether to descend. Closing and lifting become further decisions in the same language.

The useful abstraction is a small physical change with an observable consequence. A task can unfold through repeated decisions rather than requiring the model to specify the entire trajectory at once.

There is an important lineage here. Code as Policies showed that language models could generate robot programs containing control calls, calculations, conditionals, and feedback loops. That work made programming interfaces a bridge between language reasoning and physical execution. Show-Harness explores a different placement of that bridge: the model repeatedly selects incremental actions from a compact alphabet.

The frontier-model route adds a configurable harness around those decisions. Plugins organize camera information, describe proprioceptive state, maintain recent-action context, and support subtask planning. Situated planning can defer a decision until new evidence becomes available. Recovery logic can detect an empty grasp and initiate another attempt. These are explicit parts of the control system, implemented as separable modules.

The compact-model route uses supervised action imitation. The authors adapt Qwen3.5-2B with LoRA while freezing the vision tower. Training examples pair observations and instructions with the demonstrated action symbol. The released model card provides adapters for several backbone sizes trained on the same real-robot corpus, making model scale an independently adjustable part of the recipe.

A useful way to understand this design is to separate which way to move from how that movement is executed. The learning problem becomes selecting among meaningful alternatives. The robot implementation supplies the operational meaning of each alternative. That division gives the interface itself a substantial role in shaping what the policy must learn.

Data collection uses that same division. GUMI, the GUI Manipulation Interface, exposes actions through browser controls and keyboard inputs. Humans or GUI-operating agents can demonstrate behavior, with each step recorded as an observation-action pair.

The public real-robot corpus contains 164 episodes from Franka and AgileX. A separate simulation corpus combines demonstrations from ManiSkill and RoboLab. Both use the same basic action vocabulary and observation packaging.

That is the elegant systems contribution: demonstration collection, policy training, and execution share one control language. The representation is doing useful work at every stage.

The Result.

The headline number is 86 percent success for the fine-tuned compact model across the main cross-task evaluation. The strongest compared baseline reaches 57 percent. The compact policy also finishes close to Show-Harness’s frontier-powered version on this test.

The central experiment combines five objects with a plate or bowl, producing ten real-robot pick-and-place tasks. Each receives ten trials with randomized placements and a fixed action budget. A teddy bear and chess piece are held out from fine-tuning.

For the trainable VLA comparisons, π0.5 and GR00T receive continuous end-effector trajectories converted from the same demonstrations used by Show-Harness. That holds the demonstration source constant while comparing different learning and control recipes.

The engineering significance is the combination of a compact model and a deliberately structured output space. When comparing systems trained from a common demonstration source, the action representation becomes an important candidate explanation for the performance gap. Here, making the target easier to express is part of the learning strategy.

There is also a simulation-to-real result: a policy trained only on simulated demonstrations succeeds on the real Franka, while both tested trainable VLA baselines fail in that evaluation. The released simulation adapter is a single policy trained across both simulators.

This is a useful direction for practitioners because the interface creates an explicit place to standardize training and deployment. You can ask whether the camera interpretation, movement meanings, and observation format agree, instead of treating transfer as an entirely opaque property of the weights.

The most revealing ablation examines what makes the action language effective. In the zero-shot system, arbitrary action labels work almost as well as meaningful names when the prompt explicitly explains their physical effects. Written conventions provide substantial grounding; familiar names provide a useful prior.

That result makes interface specification a research variable. Choosing the vocabulary and documenting its meaning belong alongside backbone selection and fine-tuning strategy.

The compute claim is concrete: Qwen3.5-2B fine-tuning takes under two hours on an H200, and the experiments serve local models on an RTX 5090. The workflow starts from a pretrained VLM and produces a LoRA adapter that the released serving stack can load.

The Biggest Weakness.

The compact controller is not a capability-equivalent replacement for the complete frontier agent on reasoning-heavy tasks.

The paper tests searching for a hidden object and arranging letter blocks to spell “SHOW.” The standalone fine-tuned policy achieves 10 percent success, compared with 85 percent for the frontier system with situated planning. Supplying Gemini-generated subtask instructions substantially improves the compact policy.

The implementation makes the distinction clear. The released compact runtime has no separate subgoal planner or stage machinery. It directly maps the current views, task text, and recent movements to an action. The frontier route includes explicit planning support around its controller.

This matters because action imitation and complete-agent distillation are different promises. A controller can become very good at executing a specified objective without learning the process that selects the right intermediate objectives.

Putting an identified object into an identified receptacle supplies much of the task structure in the instruction. Searching behind occluders asks the system to choose informative actions and revise what happens next. The latter demands a different kind of competence from choosing the next corrective movement.

The comparison changes both the model and its surrounding machinery, so it does not establish a fundamental ceiling on small models. Its system-level implication is narrower and decisive: the evaluated standalone compact route does not reproduce the full frontier system’s reasoning performance.

A convincing capability-preserving replacement would run those reasoning tasks with an entirely local stack, explicitly account for every planning call, and compare systems with matched planning support. That would reveal whether a small planner, additional demonstrations, or a different training objective closes the gap.

Until then, “without sacrificing task capability” should refer to the demonstrated manipulation regime. The paper’s own reasoning experiments do not support extending that phrase to the complete agent.

Verdict.

Show-Harness deserves attention from teams building language-conditioned manipulation with modest demonstration budgets. Its strongest contribution is a concrete way to make the prediction problem easier while retaining a closed perception-action loop.

For an imitation-learning researcher, the interesting object is the action interface. Treat it as something to design and evaluate alongside the neural architecture. Ask which distinctions the model should learn, which conventions can be fixed explicitly, and which details belong in the robot implementation. Those choices determine what your demonstrations actually teach.

The comparison with SmolVLA is useful. SmolVLA pursues affordable robotics through a compact vision-language backbone, a learned flow-matching action expert, and asynchronous execution of action chunks. Its experiments include simulation and real robots, and its deployment design targets consumer hardware. Show-Harness contributes a complementary design point: ordinary VLM adaptation toward a small, explicit control vocabulary.

These approaches suggest a productive research question: which output representation makes the best use of the data and feedback available for your task? Model size is one part of that question. The shape of the action prediction problem is another.

The release also supplies substantial engineering material. The repository includes GUMI collectors, robot runners, plugin implementations, and the training pipeline. Published adapters and demonstration datasets provide concrete starting points for experimentation.

That makes the project valuable beyond its reported success rate. A team can investigate the interface, change the learning recipe, and study how the same demonstrations support different controller designs.

The practical takeaway is direct: make physical decisions easy to express, easy to demonstrate, and easy to interpret. Show-Harness turns that principle into a working manipulation system. For researchers deciding where to spend their next unit of complexity, the interface deserves a place beside the model.

Outro.

Watch whether this shared action language becomes a reusable training interface across more laboratories and robot platforms.

The enduring question is where to put complexity. You can put it into a larger model, a richer action decoder, a planning system, or the contract connecting perception to motion. Show-Harness makes that last choice unusually tangible.

A robot policy’s output is more than a tensor. It is an agreement about what the machine will do next. Designing that agreement can change the entire learning problem.

That is the idea worth carrying into your next experiment.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Can One Human Video Teach Vision-Based Dexterous Manipulation?

    Nov 20, 202613 minAudio

    Can a single human video provide enough guidance for a dexterous robot to learn a useful manipulation skill? This episode explains the proposed division of labor between motion priors, residual reinforcement learning,...

    Credit the people on this episode

  2. ROAM: Can Vision-Language Risk Maps Improve Legged Navigation?

    Nov 20, 202613 minAudio

    How can a legged robot use a vision-language model to navigate trails without asking it to place every foot? This episode explains how ROAM connects semantic scene interpretation to reinforcement-learned navigation and...

    Credit the people on this episode

  3. Doppel: Can Rendered Poses Improve Cross-Embodiment Robot World Models?

    Nov 19, 202613 minAudio

    Can robot world models predict interactions better if we show them how the robot will move? This episode examines Doppel's rendered-pose conditioning, its experimental evidence, and what cross-embodiment prediction...

    Credit the people on this episode

  4. Elastic-WAM: Can Selective Imagination Speed Up Robot Manipulation?

    Nov 19, 202613 minAudio

    Can a robot spend less time imagining the future and still manipulate objects more successfully? This episode explains how Elastic-WAM turns future refinement into a learned computation-allocation problem. We examine...

    Credit the people on this episode

  5. ScaleVLA: Can a Frozen 552B VLM Improve Robot Manipulation?

    Nov 18, 202613 minAudio

    Can a frozen 552-billion-parameter vision-language model make robot manipulation more data-efficient? This episode explains what ScaleVLA connects, what its experiments measure, and what its results establish about...

    Credit the people on this episode