GSP-3D: Can Gaussian World Models Improve Bimanual Manipulation?.
GSP-3D: generalizable 3D Gaussian Splatting with diffusion policy and world model for long-horizon bimanual manipulation • Xukun Liu et al. • Northwest Institute of Mechanical and Electrical Engineering; Jiangxi Guoke Defence Group Co., Ltd. • Frontiers in Neurorobotics • 2026.
Cold Open.
The left gripper has the block. The right gripper is closing around it. Now decide when the left should let go.
That instant makes bimanual manipulation interesting. Getting both arms near the object is only the beginning. The useful state includes their relationship to the object and the consequences of the next movement.
Imagine training on successful handovers. You could ask the learner to reproduce the recorded actions. You could also ask it to account for what those actions change: where the block moves, how its orientation changes, and what becomes visible as the fingers withdraw.
GSP-3D builds its story around that second question, combining Gaussian scene representations, action diffusion, and predictive learning.
The tension is architectural. Should better coordination come from a richer description of the present, or from teaching the policy about the future? And how much of that teaching needs to remain in the execution loop?.
The Idea.
Start with the closest familiar building block: 3D Diffusion Policy, or DP3. It encodes a sparse point cloud into a compact visual representation, then conditions action generation on that representation and the robot’s state. Its central design choice is to give imitation learning an explicitly three-dimensional observation space. Geometry becomes an input to the policy rather than something the policy must recover entirely from image appearance.
The action generator comes from Diffusion Policy. Instead of predicting a single deterministic command, it learns to denoise action sequences conditioned on observations. That supports multiple plausible behaviors and coordinated sequences of commands. The original framework also uses receding-horizon execution: generate a sequence, execute part of it, then update from new observations.
Gaussian splatting changes the geometric vocabulary. A Gaussian primitive has a location and a spatial footprint that can stretch and rotate. Collections of these primitives support differentiable rendering, connecting a three-dimensional representation to the images used to supervise it. The original Gaussian-splatting work optimized these scene representations for efficient, high-quality rendering. For robot learning, the interesting opportunity is to make that representation useful for decisions as well.
GSP-3D predicts Gaussians from a single RGB-D frame, aggregates them through sparse voxelization and attention, and conditions a diffusion U-Net on the resulting features, robot state, and task identity. Reconstruction supplies visual supervision.
Think about why that arrangement is attractive. In our handover example, the controller needs a representation that preserves relationships among two grippers and a manipulated object. A useful representation should make the block’s pose and the approaching fingers easy to distinguish. The design bet is that structured primitives provide a helpful intermediate description between raw sensory measurements and motor commands.
The second part of the design concerns time. A pretrained, frozen transformer predicts future Gaussian states; its prediction error regularizes policy training. This world model is absent at deployment.
That gives the predictor a teaching role. Imitation asks whether an action resembles demonstrated behavior. Predictive supervision adds another question: does the learned account of what happens next agree with the observed continuation? In a hypothetical handover, those learning signals could favor features that preserve the block’s movement and the arms’ relative arrangement across the transition.
This is a useful way to think about world models more broadly. The Gaussian World Model paper, GWM, explores both predictive representations for imitation learning and learned simulation for model-based reinforcement learning. Those are different jobs: improving what a policy learns from observations, and supplying imagined interactions for learning behavior.
For this episode, the first job is the important one. The architectural opportunity is to spend training effort on a richer account of action consequences, then let the learned policy carry the benefit into execution. The prediction task becomes a reason to preserve information that action imitation alone might not emphasize.
The Result.
The headline is a 7.75-percentage-point improvement over DP3, calculated as my unweighted average of the paper’s per-task results. The evaluation covers eight simulated Aloha-AgileX tasks, using fifty demonstrations per task, and every listed task improves.
That comparison takes place inside RoboTwin 2.0. The platform combines automated expert-data generation with a bimanual manipulation benchmark. Its data-generation machinery produces task executions in simulation, while structured randomization varies factors such as clutter, lighting, backgrounds, tabletop height, and language instructions. This makes environmental variation part of the experimental apparatus rather than a collection of manually assembled demonstrations.
Why use an unweighted task average? It answers a straightforward question: if we give each evaluated task equal importance, how much does the success rate move? For a practitioner deciding whether to investigate a representation change, that is a useful summary. It keeps the discussion attached to completed tasks instead of asking the listener to infer control quality from reconstruction images.
The comparison with DP3 is especially informative because DP3 already operates in three-dimensional observation space. The question is therefore more specific than whether depth helps an image policy. It asks what a richer geometric representation and predictive training objective can add to a policy that already receives explicit spatial information. That is the relevant engineering decision for a team with an established RGB-D imitation-learning stack.
The ablations supply the second important finding: removing either component lowers success, while removing both returns the DP3 results. The full system also leads DP3 under the reported visual-randomization conditions.
Read those ablations as an intervention on the design. If future prediction contributed nothing, there would be little reason to maintain a separate predictive-learning stage. If the scene representation contributed nothing, the extra geometric machinery would be difficult to justify. The reported pattern gives both components a role within this implementation.
The randomization experiment addresses a different engineering question: whether the advantage survives changes to the observation’s appearance. Backgrounds and illumination can change while the desired physical interaction remains similar. That makes the experiment a useful probe of whether learning is organized around the interaction rather than a particular visual arrangement. RoboTwin was designed to make those variations systematic.
Together, these results make a coherent case for investigating the combined design: richer scene conditioning helps, predictive supervision helps, and their combination produces the strongest reported policy.
The Biggest Weakness.
The biggest weakness is the missing comparison with the closest prior method. GSP-3D’s baseline lineup excludes DP4, or 4D Diffusion Policy.
Published at ICCV 2025, DP4 already combined diffusion-based action generation with a generalizable Gaussian regressor, current-scene reconstruction, and future-scene prediction. Its Gaussian world model supplies training supervision rather than running during policy inference. That is a particularly close predecessor to the central architectural argument here.
The implementations are meaningfully different. DP4 builds local point-cloud and global voxel features, and supervises predicted futures through rendered RGB-D observations. GSP-3D instead makes predicted Gaussian features part of policy conditioning and applies a prediction objective in Gaussian-parameter space. Those are concrete choices worth testing against one another.
The unanswered question is whether those choices improve on the existing Gaussian-prediction recipe.
Deleting components from your own system establishes their internal contribution. It does not establish that your particular combination is better than an earlier combination built around the same principle. An improvement over DP3 can support predictive geometric supervision without identifying which implementation of that supervision a roboticist should choose.
A matched RoboTwin comparison with DP4 would address this directly: the same demonstrations, observation access, task configurations, and evaluation protocol. The important outcome would be the difference between the two complete learning systems under shared conditions, followed by an explanation of which architectural choice drives it.
This is not evidence that DP4 would win. It is an unresolved comparison that determines the strength of GSP-3D’s advance over its closest predecessor. For a paper proposing a new integration of Gaussian perception, predictive learning, and diffusion control, that is the experiment that most directly tests the thesis.
Verdict.
GSP-3D should interest researchers working on data-efficient bimanual imitation learning, especially teams already comfortable with RGB-D observations and diffusion-based action generation.
The useful design question is where to invest additional supervision. You can put more capacity into the action generator. You can improve the scene representation. Or you can ask the training system to explain how the scene changes under actions. This paper makes the latter two choices worth examining together.
For practitioners, the conceptual payoff is a different way to value demonstration data. A demonstration is more than a collection of isolated observation-action pairs. It contains an ordered record of interaction. When designing a learner, ask what useful training targets that ordering makes available. Object movement, changing visibility, and coordinated transitions are all reasons to examine the continuation of a trajectory.
There is also a concrete software starting point for exploring this design space. DP3 has an official released implementation, and RoboTwin documents data collection, task configuration, robot control, and policy interfaces. Those resources provide established infrastructure for investigating representation and supervision choices in a shared environment.
For world-model researchers, the broader lesson is to choose the model’s job deliberately. GWM’s exploration of representation learning and imagined interaction illustrates how different that job can be across systems. A predictive model can contribute through the information it teaches an encoder to retain, through the experience it generates, or through decisions made using its forecasts.
My takeaway is straightforward: use future prediction to make the policy’s geometric representation answerable to interaction.
That is an actionable research direction. It connects perception quality to behavior, gives temporal structure in demonstrations a purpose, and makes the choice of scene representation part of the control-learning problem.
Outro.
Return to the handover. One gripper is holding the block; the other is ready to receive it.
The interesting learning question is what the policy must retain to make that transition work. Which spatial relationships matter? Which changes should it anticipate? What should a successful demonstration teach beyond the recorded commands?.
Watch for predictive supervision that improves those contact transitions consistently across manipulation tasks. That is where scene modeling becomes useful control knowledge.
This has been Embodied AI 101: looking past the rendered scene to the behavior the representation helps a robot learn.