3D-DLP: Self-supervised 3D Object-centric Scene Representation Learning
Ellina Zhang, Madhavan Iyengar, Amir Zadeh, Chuan Li, David Held, Deepak Pathak, Tal Daniel
We introduce 3D-DLP, a self-supervised object-centric representation learning model that decomposes scene-level RGB-D or voxel observations into a set of 3D latent particles. Building on the Deep Latent Particles (DLP) framework, each particle encodes disentangled attributes, including 3D keypoint position, bounding box dimensions, and appearance features, and represents a distinct entity in the scene. The model learns interpretable per-particle segmentation maps through an end-to-end self-supervised reconstruction objective. We demonstrate on both simulated and real-world datasets that the learned latent space is interpretable and controllable: by manipulating particle positions and decoding, we can generate novel scene configurations. Furthermore, we show that leveraging these compact 3D latent particles for downstream robotic manipulation improves performance over baselines that either lack explicit 3D information or rely on memory-intensive dense 3D inputs without object-centric structure. Code and videos are available at https://eubooks3003.github.io/3d-dlp
object-centric representationself-supervised learning3d representationrobotic manipulation
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 A Generalist Pair-wise Progress Critic Model for Vision-Language-Action Robots
Qi Zhang, Shaopeng Zhai, Shengzhe Zhang, Litao Liu, Tianyi Zhang, Fuxian Huang, Ming Zhou
Recent advances in Vision-Language-Action (VLA) models have significantly improved robotic perception and manipulation capabilities, but still struggling to adapt in dynamic, open-ended real-world environments due to a lack of reliable task progress feedback and improvement mechanisms. To address these challenges, we propose a generalist Vision Language Action-Critic model, VLAC, which can integrate both human and robot data, and unify action policy and task progress critic within a single autoregressive architecture. Specifically, we propose a scalable and generalizable pair-wise progress understanding approach that can predict the delta of task progress between two steps in a trajectory and generate correct actions to complete the task. Then, we trained the model on large-scale, multi-source human, robot, and general vision-language data for a generalist. Furthermore, we deploy reinforcement learning where VLAC can autonomously evaluate task progress to provide intrinsic rewards. Extensive evaluations demonstrate that our model generalizes effectively across diverse tasks and environments, leveraging its pair-wise progress understanding to provide reliable dense rewards, robust action generation, and significant improvements in real-world reinforcement learning. Our codes are available at \url{https://github.com/InternRobotics/VLAC}.
vision-language-actionreward modelreal-world rlrobot manipulation
OpenReview ↗
PDF ↗
Code ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 A Mechanistic Analysis of Sim-and-Real Co-Training in Generative Robot Policies
Yu Lei, Minghuan Liu, Abhiram Maddukuri, Zhenyu Jiang, Yuke Zhu
Co-training, which combines limited in-domain real-world data with abundant surrogate data such as simulation or cross-embodiment robot data, is widely used for training generative robot policies. Despite its empirical success, the mechanisms that determine when and why co-training is effective remain poorly understood.
We investigate the mechanism of sim-and-real co-training through theoretical analysis and empirical study, and identify two intrinsic effects governing performance. The first, "structured representation alignment", reflects a balance between cross-domain representation alignment and domain discernibility, and plays a primary role in downstream performance. The second, the "importance reweighting effect", arises from domain-dependent modulation of action weighting and operates at a secondary level.
We validate these effects with controlled experiments on a toy model and extensive sim-and-sim and sim-and-real robot manipulation experiments. Our analysis offers a unified interpretation of recent co-training techniques and motivates a simple method that consistently improves upon prior approaches.
More broadly, our aim is to examine the inner workings of co-training and to facilitate research in this direction.
sim2realco-trainingimitation learningrobot manipulation
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 ABC-Bench: An Agentic Bio-Capabilities Benchmark for Biosecurity
Andrew Bo Liu, Samira Nedungadi, Bryce Cai, Alex Kleinman, Harmon Bhasin, Seth Donoughe
Large language models (LLMs) are rapidly acquiring capabilities relevant to biological research, from literature synthesis to interpretation of experimental data. Increasingly, LLM agents can also perform *in silico* biology tasks that previously required experienced human biologists. These emerging AI capabilities offer new opportunities for scientific discovery and biomedical advances, but they also shift the landscape of biosecurity risks. To address this, we introduce the Agentic Bio-Capabilities Benchmark (ABC-Bench), a suite of tasks to measure agentic biosecurity-relevant capabilities. ABC-Bench evaluates LLM agents on both benign and dual-use biology tasks: writing code to operate liquid handling robots, designing DNA fragments for in vitro assembly, and evading DNA synthesis screening. These tasks require a combination of biology and software expertise. All tested LLM agents outperformed the median expert human baseliner on all three tasks. Agents performed highly on tasks drawing on published knowledge and well-documented protocols, and more weakly on a task requiring novel bioinformatics reasoning. In three wet-lab validation experiments, we found that OpenAI’s o4-mini-high produced scripts that, when run on an OpenTrons liquid handling robot, successfully assembled DNA with expected sequences.
biosecurityllm-agentsbenchmarklab-automationdual-use
OpenReview ↗
PDF ↗
Poster Session 8 (Hall A) · Thursday, July 9, 5:00-6:45 AdaNav: Adaptive Reasoning with Uncertainty for Vision-Language Navigation
Xin Ding, Jianyu Wei, Yifan Yang, Shiqi Jiang, Qianxi Zhang, Hao Wu, Fucheng Jia, Liang Mi, Yuxuan Yan, Weijun Wang, Yunxin Liu, Zhibo Chen, Ting Cao
Vision-Language Navigation (VLN) requires agents to follow natural language instructions by grounding them in sequential visual observations over long horizons. Explicit reasoning could enhance temporal consistency and perception–action alignment, but reasoning at fixed steps often leads to suboptimal performance and unnecessary computation. To address this, we propose AdaNav, an uncertainty-based adaptive reasoning framework for VLN. At its core is the Uncertainty-Adaptive Reasoning Block (UAR), a lightweight plugin that dynamically triggers reasoning. We introduce Action Entropy as a policy prior for UAR and progressively refine it through a Heuristics-to-RL training method, enabling agents to learn difficulty-aware reasoning policies under the strict data limitations of embodied tasks. Results show that with only 6K training samples, AdaNav achieves substantial gains over closed-source models trained on million-scale data, improving success rate by 20% on R2R val-unseen, 11.7% on RxR-CE, and 11.4% in real-world scenes.
vision-language navigationuncertainty estimationembodied agentreal-world
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Adaptive Quasimetric Mapping : Principled Topological Abstraction for Robust Offline Goal-Conditioned Navigation
Anthony Kobanda, Waris Radji, Odalric-Ambrym Maillard, Rémy Portelas
Goal-Conditioned Reinforcement Learning aims to design agents that can reach specified goals, notably from previously collected trajectories in the offline setting. In this context, graph-based approaches have been proposed to mitigate compounding value-estimation errors in long-horizon navigation tasks. However, existing approaches typically rely on dense keypoint coverage of the dataset support, resulting in computationally expensive planning. Moreover, they lack explicit mechanisms to adapt to topological changes (e.g., new obstacles), hindering deployment in live applications such as video game environments. To address these two shortcomings, we introduce Adaptive Quasimetric Mapping (AQM), an offline framework leveraging a “time-to-reach” quasimetric learned from the available data. Crucially, it builds a sparse cover of the dataset support, as a greedy approximation to a dominating set problem. At test-time, the resulting graph is carefully pruned by comparing the observed edge traversal time against a time-to-reach budget derived from the quasimetric, thus enabling zero-shot replanning. Empirically, we evaluate AQM on navigation tasks ranging from a classical to a video-game-like benchmark evaluating adaptation across tasks. We show that AQM achieves competitive performance while requiring up to 100× fewer keypoints than prior approaches, hence demonstrating the relevance of topological abstraction for goal-conditioned navigation.
goal-conditionedoffline-rltopological-mappingnavigation
OpenReview ↗
PDF ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 Adversarial Dual On-Policy Distillation from Expressive Teacher
Zhenglin Wan, Jingxuan Wu, Xingrui Yu, Chubin Zhang, Mingcong Lei, Bo An, Ivor Tsang, Yang You
Learning from demonstrations in embodied control is often cast as behavioral cloning, and recent diffusion or flow-matching policies improve this paradigm by modeling multi-modal expert actions. Yet these methods remain offline supervised learners: the policy is trained only on expert states and receives no corrective signal on the states it actually visits. On-policy distillation (OPD) offers a natural remedy, but standard OPD assumes a strong fixed teacher, which is unavailable in demonstration-only control. We propose FA-OPD, an adversarial dual on-policy distillation method in which a Flow Matching (FM) teacher is learned from demonstrations and co-trained with a lightweight MLP student. The teacher provides two complementary signals on student rollouts. The reward channel learns an expert-likeness objective over state-action pairs and drives online exploration through long-horizon policy optimization. The action channel supplies dense local targets at student-visited states, stabilizing exploitation. FA-OPD couples them so that reward distillation enables generalization beyond point-wise demonstrations, while action distillation keeps exploration anchored near expert-like behavior. Across six robot navigation, manipulation, and locomotion benchmarks, FA-OPD beats strong baselines and shows much stronger robustness under noisy or limited demonstrations. Source code: https://github.com/vanzll/FA-OPD.
on-policy distillationflow matching policyrobot manipulationlocomotionimitation learning
AIR-VLA: Vision-Language-Action Systems for Aerial Manipulation
Jianli Sun, Bin Tian, Qiyao Zhang, Chengxiang Li, Song Zihan, Zhiyong Cui, Yisheng Lv, Yonglin Tian
While Vision-Language-Action (VLA) models have achieved remarkable success in ground-based embodied intelligence, their application to Aerial Manipulation Systems (AMS) remains a largely unexplored frontier. The inherent characteristics of AMS, including floating-base dynamics, strong coupling between the UAV and the manipulator, and the multi-step, long-horizon nature of operational tasks, pose severe challenges to existing VLA paradigms designed for static or 2D mobile bases. To bridge this gap, we propose AIR-VLA, the first VLA benchmark specifically tailored for aerial manipulation. We construct a physics-based simulation environment and release a high-quality multimodal dataset comprising 3000 manually teleoperated demonstrations, covering base manipulation, object & spatial understanding, semantic reasoning, and long-horizon planning. Leveraging this platform, we systematically evaluate mainstream VLA models and state-of-the-art VLM models. Our experiments not only validate the feasibility of transferring VLA paradigms to aerial systems but also, through multi-dimensional metrics tailored to aerial tasks, reveal the capabilities and boundaries of current models regarding UAV mobility, manipulator control, and high-level planning. AIR-VLA establishes a standardized testbed and data foundation for future research in general-purpose aerial robotics. The resource of AIR-VLA will be available at https://anonymous.4open.science/r/AIR-VLA-dataset-B5CC/.
aerial manipulationuavteleoperationbenchmark
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Any3D-VLA: Enhancing VLA Robustness via Diverse Point Clouds
Xianzhe Fan, Shengliang Deng, Xiaoyang Wu, Yuxiang Lu, Zhuoling Li, Mi Yan, Yujia Zhang, Zhizheng Zhang, He Wang, Hengshuang Zhao
Existing Vision-Language-Action (VLA) models typically take 2D images as visual input, which limits their spatial understanding in complex scenes. How can we incorporate 3D information to enhance VLA capabilities? We conduct a pilot study across different observation spaces and visual representations. The results show that explicitly lifting visual input into point clouds yields representations that better complement their corresponding 2D representations. To address the challenges of (1) scarce 3D data and (2) the domain gap induced by cross-environment differences and depth-scale biases, we propose Any3D-VLA. It unifies the simulator, sensor, and model-estimated point clouds within a training pipeline, constructs diverse inputs, and learns domain-agnostic 3D representations that are fused with the corresponding 2D representations. Simulation and real-world experiments demonstrate Any3D-VLA's advantages in improving performance and mitigating the domain gap. Our project homepage is available at https://xianzhefan.github.io/Any3D-VLA.github.io.
point cloud3d representationsim-to-realrobustness
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 AutoMoT: A Unified Vision-Language-Action Model with Asynchronous Mixture -of-Transformers for End-to-End Autonomous Driving
Wenhui Huang, Songyan Zhang, Qihang Huang, Zhidong Wang, Zhiqi Mao, Collister Chua, Zhan Chen, Long Chen, Chen Lv
Integrating vision-language models (VLMs) into end-to-end (E2E) autonomous driving (AD) systems has shown promise in improving scene understanding. However, existing integration strategies suffer from several limitations: they either struggle to resolve distribution misalignment between reasoning and action spaces, underexploit the general reasoning capabilities of pretrained VLMs, or incur substantial inference latency during action policy generation, which degrades driving performance. To address these challenges, we propose AutoMoT in this work, an end-to-end AD framework that unifies reasoning and action generation within a single vision-language-action (VLA) model. Our approach leverages a mixture-of-transformer (MoT) architecture with layer-wise joint attention sharing, which preserves the general reasoning capabilities of pre-trained VLMs while enabling efficient asynchronous inference over various tasks at different frequencies. Additionally, we explore a VLA-oriented action refiner that further enhances driving performance via diffusion-based fine-tuning. Extensive experiments on multiple benchmarks, under both open- and closed-loop settings, demonstrate that AutoMoT achieves state-of-the-art (SOTA) performance compared to existing methods. We further investigate the functional boundary of pre-trained VLMs in AD, examining when and to what extent AD-tailored fine-tuning is necessary.
autonomous-drivingvlamixture-of-transformersend-to-end
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 BEAR: Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosis
Yu Qi, Haibo Zhao, Ziyu Guo, Siyuan Ma, Ziyan Chen, Yaokun Han, Renrui Zhang, Zitiantao Lin, Yizhe Zhu, Shiji Xin, Yijian Huang, Boce Hu, Kai Cheng, Jiayi Zhang, Peiheng Wang, jiazheng liu, Wenqing Wang, Yiran Qin, Haojie Huang, Lawson L.S. Wong
Understanding the capability bottlenecks of embodied multimodal large language models (MLLMs) is crucial for improvement. However, existing embodied benchmarks fail to provide actionable insights because they focus on task-level evaluation rather than discovering capability bottlenecks. To address this, we introduce BEAR, where we divide embodied tasks into 14 atomic skills for skill-level evaluation. BEAR comprises 4,469 interleaved image–video–text entries across 14 skills in 6 categories, ranging from low-level perception to high-level planning. We evaluate 20 MLLMs on BEAR under a hierarchical skill-level diagnosis framework and discover that (1) perceptual capabilities are major bottlenecks behind reasoning failures, and (2) models fail due to unstable spatiotemporal modeling which remain unexposed in previous benchmarks. Furthermore, building on these insights, we propose BEAR-Agent, a multimodal conversable agent that augments MLLMs with visual and spatial tools. It substantially enhances MLLMs’ performance across skills, yielding a relative improvement of 17.5% on GPT-5 on BEAR and outperform baselines by a large margin in both simulation and real-robot experiments across models. We provide our project website at https://bear-official66.github.io/.
embodied aiskill evaluationmultimodal llmreal-robot benchmark
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Behavioral Mode Discovery for Fine-tuning Multimodal Generative Policies
Alberta Longhini, David Emukpere, Jean-Michel Renders, Seungsu Kim
We address the problem of fine-tuning pre-trained generative policies with reinforcement learning (RL) while preserving the multimodality of their action distributions. Existing methods for RL fine-tuning of generative policies (e.g., diffusion policies) improve task performance but often collapse diverse behaviors into a single reward-maximizing mode. To mitigate this issue, we propose an unsupervised mode discovery framework that uncovers latent behavioral modes within generative policies. The discovered modes enable the use of mutual information as an intrinsic reward, regularizing RL fine-tuning to enhance task success while maintaining behavioral diversity. Experiments on robotic manipulation tasks demonstrate that our method consistently outperforms conventional fine-tuning approaches, achieving higher success rates and preserving richer multimodal action distributions.
diffusion policyrl fine-tuningmode discoveryrobotic manipulation
OpenReview ↗
PDF ↗
Poster Session 8 (Hall A) · Thursday, July 9, 5:00-6:45 Beyond Pixel Histories: World Models with Persistent 3D State
Samuel Garcin, Thomas Walker, Steven McDonagh, Tim Pearce, Hakan Bilen, Tianyu He, Kaixin Wang, Jiang Bian
Interactive world models continually generate video by responding to a user's actions, enabling open-ended generation capabilities.
However, existing models typically lack a 3D representation of the environment, meaning 3D consistency must be implicitly learned from data, and spatial memory is restricted to limited temporal context windows. This results in an unrealistic user experience and presents significant obstacles to downstream tasks such as training agents. To address this, we present PERSIST, a new paradigm of world model which simulates the evolution of a latent 3D scene: environment, camera, and renderer.
This allows us to synthesise new frames with persistent spatial memory and consistent geometry. Both quantitative metrics and a qualitative user study show substantial improvements in spatial memory, 3D consistency, and long-horizon stability over existing methods, enabling coherent, evolving 3D worlds. We further demonstrate novel capabilities, including synthesising diverse 3D environments from a single image, as well as enabling fine-grained, geometry-aware control over generated experiences by supporting environment editing and specification directly in 3D space.
world-models3d-generationvideo-generationembodied-ai
Bottleneck-Guided Spectral Subgoals For Offline Goal-Conditioned RL
Hebin Liang, Yi Ma, Chenjun Xiao, Zibin Dong, Zilin Cao, Fei Ni, Yifu Yuan, Jianye HAO
Offline goal-conditioned RL (OGCRL) learns to reach arbitrary goals from the offline dataset, but long-horizon performance hinges on crossing a handful of hard-to-cross bottlenecks. These bottlenecks not only dictate the feasible paths toward the goal but also act as critical keypoints, marking the transitions between adjacent regions and providing the agent with essential directional guidance. Prior hierarchical methods pick subgoals by time or short-horizon value heuristics, which do not localize the bottleneck, and therefore the agent loses the clear guidance that bottlenecks could provide about where to pass next. We instead model long-horizon planning as ``cross the next bottleneck'': we apply Laplacian spectral clustering to the offline dataset to expose bottlenecks, identify trajectories from the offline dataset that cross these boundaries, and use the corresponding boundary-supported states as keypoint (KP) candidates.
The most representative KPs are then instantiated from these candidates, and a directed KP reachability graph $\mathcal G_{\mathrm{KP}}$ is constructed based on the resulting KPs.
We then restrict high-level choices to these bottleneck states and use a pluggable low-level controller to execute the short transitions between them.
We provide theory showing that under a standard metastable decomposition of the state space, routing through bottlenecks yields an (approximately) optimal one-step subgoal in terms of hitting-time, and that Laplacian spectra recover bottlenecks with high overlap. Thus, Laplacian spectral clustering can discover near-optimal subgoals under these assumptions. Empirically, the same pattern holds: across D4RL and OGBench, our method achieves state-of-the-art results on a broad set of navigation and manipulation tasks and across diverse dataset regimes, for example, **96.5\%** on **AntMaze** and **84.7\%** on **Franka-Kitchen**.
offline rlgoal-conditioned rlantmazefranka-kitchensubgoal discovery
OpenReview ↗
PDF ↗
Poster Session 7 (Hall A) · Thursday, July 9, 2:30-4:15 Bring My Cup! Personalizing Vision-Language-Action Models with Visual Attentive Prompting
Sangoh Lee, Sangwoo Mo, Wook-Shin Han
While Vision-Language-Action (VLA) models generalize well to generic instructions, they struggle with personalized commands such as "bring *my* cup," where the robot must act on one specific instance among visually similar objects. We study this setting of manipulating personal objects, in which a VLA must identify and control a user-specific object unseen during training using only a few reference images. To address this challenge, we propose **Visual Attentive Prompting (VAP)**, a simple-yet-effective training-free perceptual adapter that equips frozen VLAs with top-down selective attention. VAP treats the reference images as a non-parametric visual memory, grounds the personal object in the scene through open-vocabulary detection and embedding-based matching, and then injects this grounding as a visual prompt by highlighting the object and rewriting the instruction. We construct two simulation benchmarks, Personalized-SIMPLER and Personalized-VLABench, and a real-world tabletop benchmark to evaluate personalized manipulation across multiple robots and tasks. Experiments show that VAP consistently outperforms generic policies and token-learning baselines in both success rate and correct-object manipulation, helping to bridge the gap between semantic understanding and instance-level control.
personalizationvisual promptingrobot manipulationvla
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Can VLMs Diagnose and Recover from VLA Manipulation Faults?
Bowen Yan, Jiahao Xiao, Kehui Liu, Jianbo Zhang, Zicheng Zhang, Qi Jia, Zhongjie Jia, Haoming Song, Chunyi Li, Bin Zhao, Guangtao Zhai
Existing VLA models frequently fail in robotic manipulation tasks, with poorly structured fault types that often require expert diagnosis. While VLMs offer strong explanatory capabilities, their effectiveness in assisting VLAs is limited by their unclear role in diagnostics and inadequate collaboration mechanisms. To address this, we introduce VLA-FixBench, a fault evaluation dataset that spans perception, planning, and control failures, and provides annotations for task stages, fault types, and spatiotemporal repair strategies. We further propose FaultEval, a static-to-dynamic-to-real evaluation framework that benchmarks 20 VLMs across multiple fault-related dimensions. Building on these insights, we design a VLM–VLA collaboration mechanism that localizes spatiotemporal deviations and rolls back task execution to enable targeted recovery. Experiments show that FaultEval reliably characterizes VLM-based closed-loop diagnosis and repair. The upper-bound analysis using human expert intervention shows that an idealized feedback loop can improve task success rates by 13\% on LIBERO and 35\% on real-world robots. Our code, benchmark, and project page will be publicly released at:
https://kakigo.github.io/VLA-FixBench/
fault diagnosisvlm-vla collaborationreal-robotfault recovery
CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation
Letian Fu, Justin Yu, Karim El-Refai, Ethan Kou, Haoru Xue, Huang Huang, Wenli Xiao, Li Fei-Fei, Guanya Shi, Jiajun Wu, S. Shankar Sastry, Yuke Zhu, Ken Goldberg, Linxi Fan
Code-as-Policy (CaP) is a paradigm in which a language or vision-language model generates executable robot control programs, yet its effectiveness as an autonomous controller for embodied manipulation remains underexplored. Prior CaP systems often rely on high-level, human-designed primitives, making it difficult to separate agent capability from designer-provided scaffolding. We present CaP-X, an open-access framework for systematically studying Code-as-Policy agents in robot manipulation. CaP-X includes four components. CaP-Gym is an interactive environment in which coding agents control robots by synthesizing and executing programs that compose perception and control primitives. Building on this foundation, CaP-Bench evaluates frontier language and vision-language models across varying levels of abstraction, interaction, and perceptual grounding. Across 12 models, the task success rates improve with human-crafted abstractions but degrade as these priors are removed, exposing a dependence on designer scaffolding. At the same time, we observe that scaling test-time computation with multi-turn interaction, structured execution feedback, visual differencing, automatic skill synthesis, and ensembled reasoning can substantially improve robustness even when agents operate over low-level primitives. These findings motivate CaP-Agent0, a training-free framework that achieves near human-level reliability on several manipulation tasks in simulation and on real embodiments. CaP-RL explores reinforcement learning with verifiable rewards to improve success rates and supports sim-to-real transfer through a shared code-as-action-space interface. Together, CaP-X provides an open-access platform for advancing embodied coding agents. Project page: https://capgym.github.io
code-as-policyrobot manipulationbenchmarksim-to-realllm coding agents
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Cerebellar-Inspired Residual Control for Fault Recovery: From Inference-Time Adaptation to Structural Consolidation
Nethmi Jayasinghe, Diana Gontero, Spencer T. Brown, Vinod K Sangwan, Mark C. Hersam, Amit Ranjan Trivedi
Robotic policies deployed in real-world environments often encounter post-training faults, where retraining, exploration, or system identification are impractical. We introduce an inference-time, cerebellar-inspired residual control framework that augments a frozen reinforcement learning policy with online corrective actions, enabling fault recovery without modifying base policy parameters. The framework instantiates core cerebellar principles, including high-dimensional pattern separation via fixed feature expansion, parallel microzone-style residual pathways, and local error-driven plasticity with excitatory and inhibitory eligibility traces operating at distinct time scales. These mechanisms enable fast, localized correction under post-training disturbances while avoiding destabilizing global policy updates. A conservative, performance-driven meta-adaptation regulates residual gain and plasticity, preserving nominal behavior and suppressing unnecessary intervention. Experiments on MuJoCo benchmarks under actuator, dynamic, and environmental perturbations show improvements of up to $+66$% on $\texttt{HalfCheetah-v5}$ and $+53$% on $\texttt{Humanoid-v5}$ under moderate faults, with graceful degradation under severe shifts and complementary robustness from consolidating persistent residual corrections into policy parameters.
fault-tolerant controlresidual learningrobot locomotionreinforcement learning
Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL
Jinwoo Choi, Sang-Hyun Lee, Seung-Woo Seo
Offline goal-conditioned reinforcement learning remains challenging for long-horizon tasks. While hierarchical approaches mitigate this issue by decomposing tasks, most existing methods rely on separate high- and low-level networks and generate only a single intermediate subgoal, leaving several structural limitations in long-horizon decision-making. To address this limitation, we draw inspiration from chain-of-thought reasoning and propose the Chain-of-Goals Hierarchical Policy (CoGHP), a novel framework that reformulates hierarchical decision-making as autoregressive sequence modeling within a unified architecture. Given a state and a final goal, CoGHP autoregressively generates a sequence of latent subgoals followed by the primitive action, where each latent subgoal acts as a reasoning step that conditions subsequent predictions. To implement this efficiently, we introduce an MLP-Mixer backbone, which supports cross-token communication and captures structural relationships among state, goal, latent subgoals, and action. Across challenging navigation and manipulation benchmarks, CoGHP consistently outperforms strong offline baselines, demonstrating improved performance on long-horizon tasks. Project page: https://wlsdn9350.github.io/projects/coghp/
hierarchical-rlgoal-conditionedoffline-rllong-horizon
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 Characterizing Vision-Language-Action Models across XPUs: Constraints and Acceleration for On-Robot Deployment
Kaijun Zhou, Qiwei Chen, Da Peng, Zhiyang Li, Xijun Li, Jinyu Gu
Vision-Language-Action (VLA) models are promising for generalist robot control, but on-robot deployment is bottlenecked by real-time inference under tight cost and energy budgets.
Most prior evaluations rely on desktop-grade GPUs, obscuring the trade-offs and opportunities offered by heterogeneous edge accelerators (GPUs/XPUs/NPUs).
We present a systematic framework for low-cost VLA deployment via model--hardware co-characterization.
First, we build a cross-accelerator leaderboard and evaluate model--hardware pairs under \textbf{CET} (Cost, Energy, Time), showing that ``right-sized'' edge devices can be more cost-/energy-efficient than flagship GPUs while meeting control-rate constraints.
Second, using fine-grained SM tracing and Roofline analysis, we uncover a consistent two-phase inference pattern: a compute-bound VLM backbone followed by a memory-bound Action Expert, which induces phase-dependent underutilization and hardware inefficiency.
Finally, guided by these insights, we propose \textbf{DP-Cache} and \textbf{V-AEFusion} to reduce diffusion redundancy and enable asynchronous pipeline parallelism, achieving up to \(2.9\times\) speedup on GPUs and \(6\times\) on edge NPUs with only marginal success degradation.
The example leaderboard website is: \url{https://vla-leaderboard-01.vercel.app/}.
on-robot deploymenthardware accelerationedge inferencevla
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Compositional Generative Modeling from Decentralized Data
Mashrur M. Morshed, Vishnu Boddeti
Learning the compositional nature of the physical world requires joint observation of interacting factors.
However, because practical data is often decentralized, these factors are fragmented across isolated silos.
Existing decentralized generative approaches focus only on modeling the union of siloed data, overlooking novel combinations implied by the collective whole.
To bridge this gap, we introduce Decentralized Compositional Flow Matching (DCFM), a framework that enforces structural constraints across the global set of generative factors, without exchanging any raw data. DCFM enables novel combinations to emerge through peer interactions, even when no single data source can independently support the composition. Empirically, DCFM substantially outperforms federated learning and mixture-of-experts baselines across conditional image generation, robotic spatial planning, and medical attribute co-occurrence modeling.
generative-modelingdecentralized-learningflow-matchingcompositionality
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Compositional Planning with Jumpy World Models
Jesse Farebrother, Matteo Pirotta, Andrea Tirinzoni, Marc G Bellemare, Alessandro Lazaric, Ahmed Touati
The ability to plan with temporal abstractions is central to intelligent decision-making. Rather than reasoning over primitive actions, we study agents that compose pre-trained policies as temporally extended actions, enabling solutions to complex tasks that no constituent alone can solve. Such compositional planning remains elusive as compounding errors in long-horizon predictions make it challenging to estimate the visitation distribution induced by sequencing policies. Motivated by the geometric policy composition framework introduced in Thakoor et al. (2022), we address these challenges by learning predictive models of multi-step dynamics --- so-called jumpy world models --- that capture state occupancies induced by pre-trained policies across multiple timescales in an off-policy manner. Building on Temporal Difference Flows (Farebrother et al., 2025), we enhance these models with a novel consistency objective that aligns predictions across timescales, improving long-horizon predictive accuracy. We further demonstrate how to combine these generative predictions to estimate the value of executing arbitrary sequences of policies over varying timescales. Empirically, we find that compositional planning with jumpy world models significantly improves zero-shot performance across a wide range of base policies on challenging manipulation and navigation tasks, yielding, on average, a 200% relative improvement over planning with primitive actions on long-horizon tasks.
hierarchical-planningworld-modelspolicy-compositionsuccessor-representation
OpenReview ↗
PDF ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 Compositional Transduction with Latent Analogies for Offline Goal-Conditioned Reinforcement Learning
Junseok Kim, Dohyeong Kim, Mineui Hong, Songhwai Oh
Compositional generalization is essential for reaching unseen goals under novel contextual variations in offline goal-conditioned reinforcement learning (GCRL), where a generalist goal-reaching agent must be learned from limited data. Most prior approaches pursue this via trajectory stitching over temporally contiguous segments, which limits composing behaviors across varying contexts. To overcome this limitation, we formalize *analogy transduction* as synthesizing new plans by composing task-endogenous analogies with given contexts and propose a novel analogy representation tailored for it. Grounded in our theory, this analogy representation captures what changes under optimal task execution, remains invariant to contextual variations, and is sufficient for optimal goal reaching. We further contend that generalization to unseen analogy-context pairs is a practical obstacle in analogy transduction, and introduce a new approach for offline GCRL that enables analogy transduction beyond seen pairs to unseen combinations. We empirically demonstrate the effectiveness of our approach on OGBench manipulation environments, substantially outperforming prior methods that do not perform analogy transduction.
offline rlgoal-conditioned rlogbenchcompositional generalization
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 Contrastive Representation Regularization for Vision-Language-Action Models
Taeyoung Kim, Jimin Lee, Myungkyu Koo, Dongyoung Kim, Kyungmin Lee, Changyeon Kim, Younggyo Seo, Jinwoo Shin
Vision-Language-Action (VLA) models have shown strong capabilities in robot manipulation by leveraging rich representations from pre-trained Vision-Language Models (VLMs).
However, their representations arguably remain suboptimal, lacking sensitivity to robotic signals such as control actions and proprioceptive information.
To address the issue, we introduce Robot State-aware Contrastive Loss (RS-CL), a simple and effective representation regularization for VLA models, designed to bridge the gap between VLM representations and robotic signals.
In particular, RS-CL aligns the representations more closely with the robot's proprioceptive states by using relative distances between the states as soft supervision.
Complementing the original action prediction objective, RS-CL enhances control-relevant representation learning, while being lightweight and fully compatible with standard VLA training pipelines.
Our empirical results demonstrate that RS-CL substantially improves the performance of state-of-the-art VLA models; it pushes the prior art to 69.7% achieving the state-of-the-art performance on the RoboCasa-Kitchen benchmark, and boosts success rates from 45.0% to 58.3% on challenging real-robot manipulation tasks.
contrastive learningrobot manipulationvlarepresentation learning
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning
Hexian Ni, Tao Lu, Yinghao Cai
Reward design remains a central challenge in reinforcement learning (RL). Hand-crafted rewards are often difficult to specify and may lead to suboptimal policies, while learned rewards from preferences can suffer from inefficiency and unstable training. Inspired by the dual nature of human learning explored in cognitive science, we decompose rewards into two complementary components: Formal Rewards (FR), explicitly designed based on task knowledge, and Residual Rewards (RR), learned from observations to capture implicit and nuanced preferences. Based on this decomposition, we propose CoRe, a hybrid framework that integrates FR and RR with vision-language models (VLMs) feedback to achieve preference-aligned policies without human involvement. Our contributions are twofold: (1) We propose a Formal Reward Module (FRM) that leverages VLMs to iteratively design and optimize FR based on task knowledge and preference feedback, enabling the continual improvement of policy during training; (2) We introduce a Residual Reward Module (RRM) that learns RR from video-level preference by employing VLMs to generate preference labels and capturing nuanced rewards that complement FR, ensuring alignment with human intent. Through the synergy of FRM and RRM, CoRe enables the automatic construction of reliable rewards that are efficient and preference-aligned. Extensive experiments demonstrate that CoRe outperforms existing approaches in terms of policy learning effectiveness and efficiency on ten robotic manipulation tasks in simulation and five real-worlds.
reward learningvlm feedbackrobot manipulationpreference-based rl
OpenReview ↗
PDF ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 Covariance Volume Maximization for Embodied Latent Exploration in Deep Reinforcement Learning
Yiming Wang, Yiheng Zhang, Kaiyan Zhao, Xingjie Zuo, Xingyu Liu, Xuetao Li, Furui Liu, Bo An, Leong Hou U
Efficient exploration remains a key challenge in deep reinforcement learning, especially for embodied agents operating in realistic environments with high-dimensional observations and complex dynamics. Recent latent exploration methods define bonuses in a learned latent space, but often struggle in these settings where (i) representations can be
noisy or policy-dependent, and (ii) common strategies such as randomized latent objectives or fixed directional spanning are brittle and fail to improve global coverage. We propose Covariance Volume Maximization (CVM), a coverage-driven latent exploration framework with two key components. First, we learn a behavioral state encoder using a *policy-mixture* objective to reduce representation drift under rapidly changing exploration policies, yielding stable and behaviorally meaningful latent displacements. Second, CVM rewards each transition by its exact increase in the log-determinant of the covariance of recent latent displacements, explicitly expanding the explored region and prioritizing under-covered directions. This objective coincides with the classical *D-optimal* design criterion, providing an information-efficiency justification. Extensive experiments on embodied navigation and manipulation tasks demonstrate that CVM substantially improves exploration efficiency and robustness, and scales effectively to different environments.
explorationlatent-representationembodied-rldeep-rl
OpenReview ↗
PDF ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 Cross-Embodiment Robot Foundation World Models with Latent Actions
Huang Huang, Sriram Yenamandra, Arjun Majumdar, Elie Aljalbout, Tushar Nagarajan, Tsung-Yen Yang, Akshara Rai, Michael Rabbat, Li Fei-Fei, Jiajun Wu, Tingfan Wu, Franziska Meier
The diversity of robot embodiments and action spaces makes it challenging to build robot world models that generalize across different
embodiments. We introduce the Latent Action-Conditioned Robot World Model (LAC-WM), which operates within a learned unified latent action space shared across diverse embodiments. This unified action space improves the world model’s performance when adapted to previously unseen robot embodiments. We compare LAC-WM with an Explicit Action-Conditioned World Model (EAC-WM), which conditions on explicit motion labels. Our results shows that explicit action conditioning leads to disjoint action representations across embodiments, limiting downstream performance when adapting to new robots. We evaluate both models on dexterous manipulation tasks and a modified LIBERO benchmark. LAC-WM improves downstream performance over EAC-WM by up to 46.7% on dexterous manipulation and 11.7% on LIBERO. Crucially, the unified latent action space allows LAC-WM’s downstream performance to scale positively with the number of embodiments used during pretraining. In contrast, the disjoint action space in EAC-WM leads to decreased performance as the number of pretraining embodiments increases. These results highlights the importance of a unified action space for efficient cross-embodiment learning, addressing a key challenge in robotics.
world modelcross-embodimentlatent actiondexterous manipulationlibero
Cross-Tactile Sensor Representation Learning
Yan Zhang, Zheng WANG, Pengpeng Zeng, Xing Xu, Jingkuan Song, Heng Tao Shen
Visuo-tactile sensors have been widely adopted in robotic manipulation. However, inherent heterogeneity in sensor designs hinders the learning of unified tactile representations in cross-sensor scenarios. Existing methods that focus on reconstruction or task-specific supervision often fail to capture the common information between different tactile sensors, particularly in the presence of substantial sensor variations, resulting in limited generalization to unseen sensors. To address this, we propose Cross-Tactile Sensor Representation Learning (CTSRL), a unified framework for sensor-agnostic tactile representation learning. CTSRL introduces a Cross-Sensor Modulator (CSM) to eliminate sensor-specific biases and adopts a two-stage learning paradigm: (1) leveraging aligned synthetic data for cross-sensor self-supervised learning to extract shared latent representations across sensor domains; and (2) integrating real-world multimodal tactile data to bridge the sim-to-real semantic gap through cross-modal alignment, thereby enriching representations with fine-grained semantic attributes. Experimental results show that our method demonstrates strong multi-sensor generalization, significantly improving sensor-agnostic representation learning.
tactile sensingsim-to-realrepresentation learningrobotic manipulation
CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning
Ayoub Belouadah, Sylvain KUBLER, YVES LE TRAON
Safe reinforcement learning (Safe RL) aims to maximize expected return while satisfying safety constraints, typically modeled as Constrained Markov Decision Processes (CMDPs). While primal-dual methods scale well to deep RL, they often suffer from delayed constraint correction, leading to oscillatory behavior and prolonged safety violations. In this paper, we propose *Constraint-Sensitive Policy Optimization (CSPO)*, a first-order primal-dual method that incorporates local constraint sensitivity into policy updates. CSPO augments the primal objective with a constraint-sensitive correction derived from the shortest signed distance to the safety boundary, enabling smarter recovery steps back to safety, compensating for delayed Lagrange multiplier updates, reducing oscillations near the boundary, and preserving the KKT solutions of the original constrained problem. Experiments on navigation and locomotion benchmarks demonstrate that CSPO achieves faster safety recovery and high reward preservation, resulting in higher constrained returns compared to state-of-the-art primal-dual and penalty-based methods.
safe-rlconstrained-mdppolicy-optimizationprimal-dual
OpenReview ↗
PDF ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 CURVE: Learning Causality-Inspired Invariant Representations for Robust Scene Understanding via Uncertainty-Guided Regularization
Yue Liang, Jiatong Du, Ziyi Yang, Yanjun Huang, Hong Chen
Scene graphs provide structured abstractions for scene understanding, yet they often overfit to spurious correlations, severely hindering out-of-distribution generalization. To address this limitation, we propose CURVE, a causality-inspired framework that integrates variational uncertainty modeling with uncertainty-guided structural regularization to suppress high-variance, environment-specific relations. Specifically, we apply prototype-conditioned debiasing to disentangle invariant interaction dynamics from environment-dependent variations, promoting a sparse and domain-stable topology. Empirically, we evaluate CURVE in zero-shot transfer and low-data sim-to-real adaptation, verifying its ability to learn domain-stable sparse topologies and provide reliable uncertainty estimates to support risk prediction under distribution shifts.
autonomous-drivingscene-graphsrobust-representationood-generalization
OpenReview ↗
PDF ↗
Code ↗
Poster Session 6 (Hall A) · Thursday, July 9, 10:30-12:15 DADP: Domain Adaptive Diffusion Policy
Pengcheng Wang, Qinghang Liu, Haotian Lin, Yiheng Li, Guojian Zhan, Masayoshi Tomizuka, Yixiao Wang
Learning domain adaptive policies that can generalize to unseen transition dynamics, remains a fundamental challenge in learning-based control.
Substantial progress has been made through domain representation learning to capture domain-specific information, thus enabling domain-aware decision making.
We analyze the process of learning domain representations through dynamical prediction and find that selecting contexts adjacent to the current step causes the learned representations to entangle static domain information with varying dynamical properties.
Such mixture can confuse the conditioned policy, thereby constraining zero-shot adaptation.
To tackle the challenge, we propose DADP (Domain Adaptive Diffusion Policy), which achieves robust adaptation through unsupervised disentanglement and domain-aware diffusion injection.
First, we introduce Lagged Context Dynamical Prediction, a strategy that conditions future state estimation on a historical offset context; by increasing this temporal gap, we unsupervisedly disentangle static domain representations by filtering out transient properties.
Second, we integrate the learned domain representations directly into the generative process by biasing the prior distribution and reformulating the diffusion target.
Extensive experiments on challenging benchmarks across locomotion and manipulation demonstrate the superior performance, and the generalizability of DADP over prior methods.
More visualization results are available on the https://outsider86.github.io/DomainAdaptiveDiffusionPolicy/.
diffusion policydomain adaptationrobot manipulationlocomotion
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 1 (Hall A) · Tuesday, July 7, 10:30-12:15 DDP-WM: Disentangled Dynamics Prediction for Efficient World Models
Shicheng Yin, Kaixuan Yin, Weixing Chen, Yang Liu, Guanbin Li, Liang Lin
World models are essential for autonomous robotic planning. However, the substantial computational overhead of existing dense Transformer-based models significantly hinders real-time deployment. To address this efficiency-performance bottleneck, we introduce DDP-WM, a novel world model centered on the principle of Disentangled Dynamics Prediction (DDP). We hypothesize that latent state evolution in observed scenes is heterogeneous and can be decomposed into sparse primary dynamics driven by physical interactions and secondary context-driven background updates. DDP-WM realizes this decomposition through an architecture that integrates efficient historical processing with dynamic localization to isolate primary dynamics. By employing a cross-attention mechanism for background updates, the framework optimizes resource allocation and provides a smooth optimization landscape for planners. Extensive experiments demonstrate that DDP-WM achieves superior efficiency and performance across diverse tasks, including navigation, precise tabletop manipulation, and complex deformable or multi-body interactions. Specifically, on the challenging Push-T task, DDP-WM achieves an approximately 9 times inference speedup and improves the MPC success rate from 90% to 98% compared to state-of-the-art dense models.
world modelrobot manipulationpush-tplanning efficiency
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 1 (Hall A) · Tuesday, July 7, 10:30-12:15 De4D-SLAM: Gradient-Isolated Static-Dynamic Decoupling for Monocular SLAM in Dynamic Environments
Zhicheng Fan, Zitong Wu, Zhaoxing Fan, Xiao Zhang, Biao Hou, Bo REN
Conventional dynamic SLAM approaches typically treat dynamic objects as outliers based on pre-defined categories, creating perceptual blind spots that limit the comprehensive environmental perception required for embodied agents. Although integrating Gaussian Splatting into SLAM enables holistic scene representation, it introduces an optimization paradox: without categorical priors, flexible dynamic primitives rapidly overfit static residuals. This phenomenon undermines the self-supervised error signals necessary for distinguishing motion. In response, we present De4D-SLAM, a novel framework designed for decoupled 4D reconstruction from monocular video. Our approach features a Gradient-Isolated Decoupling strategy, which leverages static reconstruction residuals to supervise a Spatially-Aware Kolmogorov-Arnold Network (SA-KAN), ensuring robust, category-agnostic motion segmentation. Additionally, we propose a Flow-Induced Initialization prior to stabilize the non-convex optimization of 4D Gaussian primitives using dense optical flow. Extensive evaluations on the TUM and Bonn benchmarks demonstrate that De4D-SLAM achieves competitive performance in both tracking and dynamic reconstruction, successfully reconciling the tension between robust localization and high-fidelity 4D mapping.
slamdynamic scene reconstructiongaussian splattingembodied perception
Debate2Create: Robot Co-design via Multi-Agent LLM Debate
Kevin Qiu, Marek Cygan
We introduce Debate2Create (D2C), a multi-agent LLM framework that formulates robot co-design as structured, iterative debate grounded in physics-based evaluation. A design agent and control agent engage in a thesis-antithesis-synthesis loop, while criterion-specific LLM judges provide multi-objective feedback to steer exploration. Across five MuJoCo locomotion benchmarks, D2C achieves the highest default-normalized score among the evaluated LLM-based and black-box baselines, with gains up to 3.2x on Ant and nearly 9x on Swimmer. Iterative debate yields 18-35% gains over compute-matched zero-shot generation, and D2C-generated rewards transfer to default morphologies in 4/5 tasks. These results suggest that structured, simulator-grounded multi-agent interaction is a useful mechanism for joint morphology-reward optimization under a fixed-topology, per-candidate-RL protocol. Project page: debate2create.github.io.
robot co-designmorphology optimizationllm agentsmujoco locomotion
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter
Xukun Li, Yu Sun, Lei Zhang, Bo-Sheng Huang, Yibo Peng, Yuan Meng, Haojun Jiang, Shaoxuan Xie, Guocai Yao, Alois Knoll, Zhenshan Bing, Xinlong Wang, Zhenguo Sun
Bimanual dexterous manipulation relies on integrating multimodal inputs to perform complex real-world tasks. To address the challenges of effectively combining these modalities, we propose DECO, a decoupled multimodal diffusion transformer that disentangles vision, proprioception, and tactile signals through specialized conditioning pathways, enabling structured and controllable integration of multimodal inputs, with a lightweight adapter for parameter-efficient injection of additional signals.
Alongside DECO, we release DECO-50 dataset for bimanual dexterous manipulation with tactile sensing, consisting of 50 hours of data and over 5M frames, collected via teleoperation on real dual-arm robots.
We train DECO on DECO-50 and conduct extensive real-world evaluation with over 2,000 robot rollouts. Experimental results show that DECO achieves the best performance across all tasks, with a 72.25\% average success rate and a 21\% improvement over the baseline. Moreover, the tactile adapter brings an additional 10.25\% average success rate across all tasks and a 20\% gain on complex contact-rich tasks while tuning less than 10\% of the model parameters.
bimanual manipulationdexterous manipulationtactile sensingdiffusion policyteleoperation
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Decompose and Recompose: Reasoning New Skills from Existing Abilities for Cross-Task Robotic Manipulation
Xitie Zhang, Aming WU, Yahong Han
Cross-task generalization is a core challenge in open-world robotic manipulation, and the key lies in extracting transferable manipulation knowledge from seen tasks. Recent in-context learning approaches leverage seen task demonstrations to generate actions for unseen tasks without parameter updates. However, existing methods provide only low-level continuous action sequences as context, failing to capture composable skill knowledge and causing models to degenerate into superficial trajectory imitation. We propose Decompose and Recompose, a skill reasoning framework using atomic skill-action pairs as intermediate representations. Our approach decomposes seen demonstrations into interpretable skill--action alignments, enabling the model to recompose these skills for unseen tasks through compositional reasoning. Specifically, we construct a task-adaptive dynamic demonstration library via visual-semantic retrieval combined with skill sequences from a planning agent, complemented by a coverage-aware static library to fill missing skill patterns. Together, these yield skill-comprehensive demonstrations that explicitly elicit compositional reasoning for skill composition and execution ordering. Experiments on the AGNOSTOS benchmark and real-world environments validate our method's zero-shot cross-task generalization capability.
robotic manipulationcross-task generalizationin-context learningskill composition
OpenReview ↗
PDF ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 DecoVer: A Decompose-and-Verify Neuro-Symbolic Framework for Embodied Task Planning with BC+
YiXiang Jiang, Binqian Xu, Xiangbo Shu
Despite their remarkable general capabilities, Large Language Models (LLMs) struggle with the precise grounding required for embodied task planning. To bridge this gap, neuro-symbolic approaches have emerged, leveraging action languages like BC+ for their formal expressiveness and reasoning flexibility. However, prior methods that naively couple LLMs with BC+ typically depend on one-shot program generation, which is brittle in dynamic environments and prone to sequential omission and causal inconsistency. To address these limitations, we propose DecoVer, a Decompose-and-Verify neuro-symbolic framework that systematically adapts BC+ to embodied task planning. Specifically, DecoVer employs a cascading decomposition strategy to partition complex knowledge into hierarchical subspaces and integrates a dual verification mechanism for syntactic and executable correctness. Extensive experiments demonstrate that DecoVer consistently outperforms LLM-based baselines across the majority of evaluation metrics, achieving a 12.9% success rate gain over the highly capable Gemini-3-Pro-Preview and a 60.9% improvement over GPT-5.1 on logically complex test cases.
embodied-planningneuro-symbolicllmtask-planning
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Demystifying Action Space Design for Robotic Manipulation Policies
Yuchun Feng, Jinliang Zheng, Zhihao Wang, Dongxiu Liu, Jianxiong Li, Jiangmiao Pang, Tai Wang, Xianyuan Zhan
The specification of the action space plays a pivotal role in imitation-based robotic manipulation policy learning, fundamentally shaping the optimization landscape of policy learning. While recent advances have focused heavily on scaling training data and model capacity, the choice of action space remains guided by ad-hoc heuristics or legacy designs, leading to an ambiguous understanding of robotic policy design philosophies. To address this ambiguity, we conducted a large-scale and systematic empirical study, confirming that the action space does have significant and complex impacts on robotic policy learning. We dissect the action design space along temporal and spatial axes, facilitating a structured analysis of how these choices govern both policy learnability and control stability. Based on 13,000+ real-world rollouts on a bimanual robot and evaluation on 500+ trained models over four scenarios, we examine the trade-offs between absolute vs. delta representations, and joint-space vs. task-space parameterizations. Our large-scale results suggest that properly designing the policy to predict delta actions consistently improves performance, while joint-space and task-space representations offer complementary strengths, favoring control stability and generalization, respectively.
action space designimitation learningbimanual robotempirical study
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Designing Observation and Action Models for Efficient Reinforcement Learning with LLMs
Deunsol Yoon, Sunghoon Hong, Whiyoung Jung, Junseok Park, Geon-Hyeong Kim, Woohyung Lim, Soonyoung Lee, Seung Hwan Kim, Kanghoon Lee
Large Language Models (LLMs) have emerged as powerful tools for semantic reasoning, enabling the formalization of tasks that traditionally relied on manual human intuition. This capability extends to environment design in reinforcement learning. While prior research predominantly focuses on reward design, the design of observation and action spaces remains relatively underexplored. We propose LLM-based design of Observation and Action Models (LOAM), a framework leveraging LLMs to construct refined agent spaces from raw environments. To mitigate the computational burden of identifying the best candidate model from stochastic LLM outputs, LOAM incorporates a continuous racing mechanism that dynamically allocates resources to prioritize the most promising configurations without additional training overhead. Empirical evaluations on HumanoidBench and Isaac Lab demonstrate that LOAM consistently outperforms handcrafted baselines in both learning speed and asymptotic performance.
humanoidobservation space designaction space designLLM-guided RLIsaac Lab
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 DexMachina: Functional Retargeting for Bimanual Dexterous Manipulation
Zhao Mandi, Yifan Hou, Dieter Fox, Yashraj Narang, Ajay Mandlekar, Shuran Song
We study the problem of functional retargeting: learning dexterous manipulation policies to track object states from human hand-object demonstrations. We focus on long-horizon, bimanual tasks with articulated objects, which are challenging due to large action space, spatiotemporal discontinuities, and the embodiment gap between human and robot hands. We propose DexMachina, a novel curriculum-based algorithm: the key idea is to use virtual object controllers with decaying strength: an object is first driven automatically towards its target states, such that the policy can gradually learn to take over under motion and contact guidance. We release a simulation benchmark with a diverse set of tasks and dexterous hands, and show that DexMachina significantly outperforms baseline methods. Our algorithm and benchmark enable a functional comparison for hardware designs, and we present key findings informed by quantitative and qualitative results. With the recent surge in dexterous hand development, we hope this work will provide a useful platform for identifying desirable hardware capabilities and lower the barrier for contributing to future research. Videos and more at \url{project-dexmachina.github.io}
dexterous manipulationbimanualretargetingcurriculum learningsimulation
DF-ExpEnse: Diffusion Filtered Exploration for Sample Efficient Finetuning
Calvin Luo, Chen Sun, Shuran Song
A natural recipe for intelligent robotic decision-making is initializing from pretrained generative control policies, which have summarized offline experience, and adapting them to self-collected online experience. We present DF-ExpEnse, an exploration technique that improves the quality of online experience collection, thus increasing finetuning sample-efficiency. DF-ExpEnse leverages the multimodal modeling capabilities of the generative control policy to create an expressive and tractably evaluatable candidate set. It then utilizes an ensemble of critics to identify the action that best balances quality with high exploration interest. In fleet settings, DF-ExpEnse further enables cross-agent communication to facilitate collaborative exploration as a group. DF-ExpEnse can be seamlessly integrated with existing strategies that finetune pretrained generative control policies via reinforcement learning. We experimentally validate consistent sample-efficiency benefits through DF-ExpEnse across a variety of manipulation and locomotion tasks, compared to default finetuning and alternative action selection schemes. Project can be found at [df-expense.github.io](https://df-expense.github.io).
explorationfleet learningmanipulationlocomotionsample-efficient RL
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Differentiable Weightless Controllers: Learning Logic Circuits for Continuous Control
Fabian Kresse, Christoph H. Lampert
Controlling autonomous systems under real-world conditions often requires policies that can be evaluated with low latency and minimal energy consumption. Unfortunately, these conditions are at odds with the use of high-precision deep neural networks as controllers. In this work, we introduce Differentiable Weightless Controllers (DWCs), a symbolic-differentiable architecture that learns flexible, non-linear, yet highly efficient control policies. DWCs can be trained end-to-end via gradient-based techniques, yet compile directly into FPGA-compatible circuits with few- or even single-clock-cycle latency and nanojoule-level energy cost per action. Across five MuJoCo benchmarks, including high-dimensional Humanoid, DWCs achieve returns competitive with standard deep policies (full-precision or quantized neural networks). Furthermore, DWCs exhibit structurally sparse and interpretable connectivity patterns, enabling direct inspection of which input values influence control decisions.
continuous-controlfpgaenergy-efficientlogic-circuits
OpenReview ↗
PDF ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 DiscoForcing: A Unified Framework for Real-Time Audio-Driven Character Control with Diffusion Forcing
Kaiyang Ji, Bingsheng Qian, Binghuan Wu, Kangyi Chen, Ye Shi, Jingya Wang
We study real-time audio-responsive character control as a deployment-faithful problem: strictly causal, bounded-latency streaming that must generate coherent full-body motion at interactive frame rates while the audio condition can change abruptly (tempo shifts, drops, or user edits). Prior music-to-motion systems are largely optimized for offline generation with global context, and degrade in streaming rollouts where conditioning history becomes stale or unreliable. We introduce DiscoForcing, a streaming audio-driven diffusion framework that combines a causal music encoder that captures rhythmic structure and phase dynamics with a diffusion-forcing sequence model trained under heterogeneous noise levels across the temporal horizon. Building on this, we design a hybrid temporal schedule and a history-guided streaming sampler to explicitly trade off responsiveness against long-horizon consistency under non-stationary audio. Implemented in an end-to-end real-time interactive system with online avatar playback and humanoid deployment workflows, DiscoForcing delivers more stable long-horizon rollouts and sharper audio–motion alignment than prior baselines under matched causality and latency constraints while maintaining real-time throughput.Project Page: https://discoforcing.github.io
character-controlmotion-generationdiffusionaudio-driven
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 Discrete Diffusion VLA: Bringing Discrete Diffusion to Action Decoding in Vision-Language-Action Policies
Zhixuan Liang, Yizhuo Li, Tianshuo Yang, Chengyue Wu, Sitong Mao, Liuao Pei, Tian Nian, Shunbo Zhou, Xiaokang Yang, Jiangmiao Pang, Yao Mu, Ping Luo
Vision–Language–Action (VLA) models adapt large vision–language backbones to map images and instructions into robot actions. However, prevailing VLAs either generate actions autoregressively in a fixed left-to-right order with poor performance or attach separate diffusion heads outside the backbone that fragments information pathways and hinders unified, scalable architectures. Instead, we present Discrete Diffusion VLA that discretizes action chunks and models them with discrete diffusion pattern retaining progressive refinement inside the unified transformer backbone. Our method achieves an adaptive decoding order that resolves high-confidence action elements before harder ones and employs secondary re-masking to revisit uncertain predictions, enabling robust error correction. This design preserves pretrained vision-language priors, supports parallel decoding, and improves the efficiency. Discrete Diffusion VLA achieves 96.4% avg. success on LIBERO, 71.2% visual matching on SimplerEnv-Fractal, and 54.2% overall on SimplerEnv-Bridge. On out-of-distribution tests of LIBERO-Goal, our method exhibits only 0.8% language degradation versus 8.0% of parallel decoding, and 20.4% vision degradation versus 29.0% for continuous diffusion, demonstrating well retention of pretrained vision-language capabilities. We also conduct two real-robot evaluations on AgileX Cobot Magic platform to show the method's effectiveness.
vision-language-actiondiscrete diffusionreal-robotLIBERO
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Dismantling the Illusion of Vision-Language-Action Models Competence via Explicit Distributional Shifts
Xueyang Zhou, Yangming Xu, Guiyao Tie, Chaoran Hu, Bo Tao, xingwei zhao, Xiang Xiang, Pan Zhou, Lichao Sun, Yongchao Chen
Given that simulation can never exhaustively enumerate reality, generalization is the determining factor for whether Vision-Language-Action (VLA) models can translate benchmark success into real-world functionality. However, current evaluation protocols often incentivize mechanical memorization rather than robust policy learning, leading to a paradoxical duality of failure: high-scoring models exhibit *spurious invariance* to semantic changes while simultaneously displaying *extreme brittleness* to trivial environmental perturbations. To address this, we introduce **LIBERO-Gen**, a diagnostic benchmark systematically designed to shift evaluation from intuition-driven heuristics to explicit distributional assumptions. Through a hierarchical protocol spanning **In-distribution**, **Compositional**, and **Domain Generalization**, LIBERO-Gen reveals performance stratifications previously masked by standard metrics. Our analysis identifies Pi0.5 as the top performer (64.0% in Spatial-CG; 21.2% in Task-CG). By identifying perceptual instability and action binding collapse as primary failure modes while validating the efficacy of structured ``Stair” sampling, LIBERO-Gen establishes a rigorous baseline for deployment reliability.
generalizationbenchmarkdistribution shiftLIBERO
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Distinguishing Imitation Error from Intrinsic Motion Learning Difficulty
Zhaorui Meng, Lu Yin, Xinrui Chen, Chengxu Zuo, Anjun Chen, Shihui Guo, Yipeng Qin
Physics-based motion imitation is central to humanoid control, yet current evaluation metrics(e.g., MPJPE) only quantify imitation outcomes, not their underlying causes. This conflation obscures a critical diagnostic question: when imitation error occurs, does it stem from policy limitations or the intrinsic learning difficulty of the target motion? To resolve this ambiguity, we propose the Torque Variation Score (TVS), a physics-grounded metric that quantifies the inherent learning difficulty of a motion independently of any policy's performance. TVS measures the magnitude of torque variation required to correct small pose perturbations, directly capturing how dynamical properties shape the reinforcement learning landscape. We establish that high-TV motions induce flat reward landscapes and vanishing policy gradients, explaining persistent imitation failures. Extensive experiments with state-of-the-art methods (UHC, PHC+) confirm TVS strongly correlates with imitation error and enables principled error attribution: high error on low-TV motions indicates policy deficiency, while high error on high-TV motions reflects fundamental learning constraints. Beyond error diagnosis, TVS facilitates three practical applications: Maximum Imitable Difficulty (MID) for policy capability assessment, Difficulty-Stratified Joint Error (DSJE) for granular performance profiling, and Flawed Motion Detection for identifying segments with abnormally high learning difficulty to support mocap data curation and quality control. TVS provides a rigorous lens to distinguish policy-induced errors from motion-inherent challenges and enhances motion dataset reliability.
motion-imitationhumanoid-controlphysics-basedevaluation-metric
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Distributional Active Inference
Abdullah Akgül, Gulcin Baykal, Manuel Haussmann, Mustafa Mert Çelikok, Melih Kandemir
Optimal control of complex environments with robotic systems faces two complementary and intertwined challenges: efficient organization of sensory state information and far-sighted action planning. Because the reinforcement learning framework addresses only the latter, it tends to deliver sample-inefficient solutions. Active inference is the state-of-the-art process theory that explains how biological brains handle this dual problem. However, its applications to artificial intelligence have thus far been limited to extensions of existing model-based approaches. We present a formal abstraction of reinforcement learning algorithms that spans model-based, distributional, and model-free approaches. This abstraction seamlessly integrates active inference into the distributional reinforcement learning framework, making its performance advantages accessible without transition dynamics modeling.
active-inferencedistributional-rlcontrolmodel-based
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 Distributional Inverse Reinforcement Learning
Feiyang Wu, Ye Zhao, Anqi Wu
We propose a distributional framework for offline Inverse Reinforcement Learning (IRL) that jointly models uncertainty over reward functions and full distributions of returns. Unlike conventional IRL approaches that recover a deterministic reward estimate or match only expected returns, our method captures richer structure in expert behavior, particularly in learning the reward distribution, by minimizing first-order stochastic dominance (FSD) violations and thus integrating distortion risk measures (DRMs) into policy learning, enabling the recovery of both reward distributions and distribution-aware policies. This formulation is well-suited for behavior analysis and risk-aware imitation learning. Theoretical analysis show that the algorithm converge with $\mathcal{O}(\varepsilon^{-2})$ iteration complexity. Empirical results on synthetic benchmarks, real-world neurobehavioral data, and MuJoCo control tasks demonstrate that our method recovers expressive reward representations and achieves state-of-the-art imitation performance.
inverse-rldistributional-rlreward-learningmujoco
OpenReview ↗
PDF ↗
Poster Session 7 (Hall A) · Thursday, July 9, 2:30-4:15 Dive into the Scene: Breaking the Perceptual Bottleneck in Vision-Language Decision Making via Focus Plan Generation
Boyuan Xiao, Bohong Chen, Yumeng Li, Ji Feng, Yao-Xiang Ding, Kun Zhou
In embodied vision-language decision making tasks such as robotic manipulation and navigation, Vision-Language and Vision-Language-Action Models (VLMs \& VLAs) are powerful tools with different benefits: VLMs are better at long-term planning, while VLAs are better at reactive control. However, their performance is limited by the same perceptual bottleneck: visual hallucinations arise due to the models’ inability to distinguish task-relevant objects from distractors. In principle, accurate identification and focus on critical objects while filtering out irrelevant ones is the key to break this limitation. A straightforward solution is one-step focus: directly attending to essential objects. However, this approach proves ineffective because effective focus inherently requires deep scene understanding. To this end, we propose ${\it SceneDiver}$, a coarse-to-fine focus plan generation method for VLMs leveraging their long-term planning abilities, that first constructs a holistic scene graph to establish initial comprehension, then progressively decomposes the task into simpler sub-problems through an iterative cycle of recognition, understanding, and analysis. To enable reactive control, we also design a lightweight adapter for distilling the deliberate focus ability into VLAs. Evaluations on standard embodied AI benchmarks confirm that our method substantially reduces visual hallucinations for both VLMs and VLAs, while preserving computational efficiency in tasks requiring fast execution. Our code and data are released at: https://future-item.github.io/SceneDiver.
vision-language modelVLAscene understandingmanipulationnavigation
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 DLO-Lab: Benchmarking Deformable Linear Object Manipulations with Differentiable Physics
Junyi Cao, Yian Wang, Ziyan Xiong, Chunru Lin, Zhehuan Chen, Chuang Gan
We address the challenge of enabling robots to manipulate deformable linear objects (DLOs), such as ropes, cables, and rubber bands. Prior work has primarily focused on narrow, task-specific problems, often relying on real-world demonstrations or handcrafted heuristics. Such approaches, however, struggle to scale to the wide variety of materials and tasks encountered in practice, and collecting sufficiently diverse real-world data is often impractical. Additionally, existing simulation environments offer limited support for the broad spectrum of material behaviors necessary for generalizable DLO manipulation. To overcome these limitations, we introduce a differentiable simulator explicitly designed for versatile DLO manipulation. Our simulator models a wide range of material properties—including (in)extensibility, elasticity, bending plasticity, and complex interactions with other objects—providing a robust foundation for learning and evaluating manipulation skills. Building on this simulator, we propose a benchmark suite of representative tasks that highlight the unique challenges of DLO manipulation. The successful execution of these tasks is often hindered by the topological complexity and grasp sensitivity inherent to DLOs. Therefore, we introduce a specialized DLO agent that explicitly manages these challenges by proposing strategic grasping points and decomposing long-horizon tasks to maximize control authority. Finally, we evaluate various policy-learning algorithms using our framework, alongside sim-to-real transfer experiments, demonstrating our platform's potential to advance DLO manipulation. Project page: https://dlo-lab-26.github.io/.
deformable object manipulationdifferentiable simulationsim2realbenchmark
DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos
Shenyuan Gao, William Liang, Kaiyuan Zheng, Ayaan Naveed Malik, Seonghyeon Ye, Sihyun Yu, Wei-Cheng Tseng, Yuzhu Dong, Kaichun Mo, Chen-Hsuan Lin, Jiannan Xiang, Yuqi Xie, Ruijie Zheng, Dantong Niu, Pooya Jannaty, Jinwei Gu, Jun Zhang, Jitendra Malik, Pieter Abbeel, Ming-Yu Liu, Yuke Zhu, Joel Jang, Linxi Fan
Being able to simulate the outcomes of actions in varied environments will revolutionize the development of generalist agents at scale. However, modeling these world dynamics, especially for dexterous robotics tasks, poses significant challenges due to limited data coverage and scarce action labels. As an endeavor towards this end, we introduce DreamDojo, a foundation world model that learns diverse interactions and dexterous controls from 44k hours of egocentric human videos. Our data mixture represents the largest video dataset to date for world model pretraining, spanning a wide range of daily scenarios with diverse objects and skills. To address the scarcity of action labels, we introduce continuous latent actions as unified proxy actions, enhancing interaction knowledge transfer from unlabeled videos. After post-training on small-scale target robot data, DreamDojo demonstrates a strong understanding of physics and precise action controllability. We also devise a distillation pipeline that accelerates DreamDojo to a real-time speed of 10.93 FPS and further improves consistency to the context. Our work enables several important applications based on generative world models, including live teleoperation, policy evaluation, and model-based planning. Systematic evaluation on multiple challenging out-of-distribution (OOD) benchmarks verifies the significance of our method for simulating open-world, contact-rich tasks, paving the way for general-purpose robot world models.
robot world modelhuman video pretrainingdexterous manipulationteleoperationlatent actions
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 DriveWorld-VLA: Unified Latent-Space World Modeling with Vision–Language–Action for Autonomous Driving
Feiyang Jia, Lin Liu, Ziying Song, Caiyan Jia, Hangjun Ye, Xiaoshuai Hao, Long Chen
End-to-end (E2E) autonomous driving has recently attracted increasing interest in unifying Vision–Language–Action (VLA) with World Models to enhance decision-making and forward-looking imagination. However, existing methods fail to effectively unify future scene evolution and action planning within a single architecture due to inadequate sharing of latent states, limiting the impact of visual imagination on action decisions. To address this limitation, we propose DriveWorld-VLA, a novel framework that unifies world modeling and planning within a latent space by tightly integrating VLA and world models at the representation level, which enables the VLA planner to benefit directly from holistic scene-evolution modeling and reducing reliance on dense annotated supervision. Additionally, DriveWorld-VLA incorporates the latent states of the world model as core decision-making states for the VLA planner, facilitating the planner to assess how candidate actions impact future scene evolution. By conducting world modeling entirely in the latent space, DriveWorld-VLA supports controllable, action-conditioned imagination at the feature level, avoiding expensive pixel-level rollouts. Extensive open-loop and closed-loop evaluations demonstrate the effectiveness of DriveWorld-VLA, which achieves state-of-the-art performance with 91.3 PDMS on NAVSIMv1, 86.8 EPDMS on NAVSIMv2, and 0.16 3-second average collision rate on nuScenes. Code and models are released at https://github.com/liulin815/DriveWorld-VLA.
autonomous drivingworld modelvision-language-actionplanning
OpenReview ↗
PDF ↗
Code ↗
Poster Session 1 (Hall A) · Tuesday, July 7, 10:30-12:15 Dual Quaternion SE(3) Synchronization with Recovery Guarantees
Jianing Zhao, Linglingzhi Zhu, Anthony Man-Cho So
Synchronization over the special Euclidean group $\mathrm{SE}(3)$ aims to recover absolute poses from noisy pairwise relative transformations and is a core primitive in robotics and 3D vision. Standard approaches often require multi-step heuristic procedures to recover valid poses, which are difficult to analyze and typically lack theoretical guarantees. This paper adopts a dual quaternion representation and formulates $\mathrm{SE}(3)$ synchronization directly over the unit dual quaternion. A two-stage algorithm is developed: A spectral initializer computed via the power method on a Hermitian dual quaternion measurement matrix, followed by a dual quaternion generalized power method (DQGPM) that enforces feasibility through per-iteration projection. The estimation error bounds are established for spectral estimators, and DQGPM is shown to admits a finite-iteration error bound and achieves linear error contraction up to an explicit noise-dependent threshold. Experiments on synthetic benchmarks and real-world multi-scan point-set registration demonstrate that the proposed pipeline improves both accuracy and efficiency over representative matrix-based methods.
se3pose-estimationslamspectral-methods
OpenReview ↗
PDF ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Model
John Won, Kyungmin Lee, Huiwon Jang, Dongyoung Kim, Jinwoo Shin
Augmenting Vision-Language-Action models (VLAs) with world models is promising for robotic policy learning but faces challenges in jointly predicting states and actions due to the modality gap. To address this, we propose DUal-STream diffusion (DUST), a world-model augmented VLA framework featuring a multimodal diffusion transformer that maintains separate modality streams while enabling cross-modal knowledge sharing. In addition, DUST utilizes independent noise perturbations and a decoupled flow matching loss to learn cross-modal causal relationships. We further introduce an asynchronous sampling method for action and vision tokens that enhances performance through inference-time scaling. Experimental results on simulated benchmarks like RoboCasa and GR-1 show that DUST achieves up to 6\% gains over state-of-the-art VLA and world-modeling baselines, with inference-time scaling providing an additional 2–5\% improvement. In real-world tasks using the Franka Research 3, DUST outperforms baselines by 10\% in success rate. Finally, we demonstrate that DUST enables effective transfer learning through both pretraining on action-free videos and joint-training with heterogeneous robot and human datasets.
world modelVLAdiffusionreal-robot manipulation
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 DyGRO-VLA: Cross-Task Scaling of Vision–Language–Action Models via Dynamic Grouped Residual Optimization
Sixu Lin, Yunpeng Qing, Litao Liu, Ming Zhou, Ruixing Jin, Xiaoyi Fan, Guiliang Liu
Recent progress in Reinforcement Learning (RL) provides a principled approach to optimizing Vision-Language-Action (VLA) models, facilitating a shift from trajectory imitation to active learning in the task environment. Despite improvements in control precision, most RL optimizers remain task-specific, which reduces VLA models from generalist controllers to policies that overfit to a narrow set of tasks. In this study, we conduct an in-depth analysis of this phenomenon and highlight the importance of cross-task feature representations for improving the generalizability of VLA models. Motivated by this finding, we introduce DyGRO-VLA, a two-stage optimization framework that 1) effectively captures cross-task latent representations based on information-theoretic principles, and 2) dynamically refines policy optimization via a mixture-of-RL-residuals. DyGRO-VLA enables the RL optimizer to exploit task-relevant latent information while strategically mitigating adverse interference on the learned representations throughout the optimization process. We evaluate our approach on LIBERO, RoboTwin2 benchmarks, and further validate it on real world, demonstrating consistent improvements over strong baselines under multi-task training and distribution shift. Our project page is available at DyGRO-VLA.
VLAreinforcement learningLIBERORoboTwinreal-world
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Dyn-VPP: Video Prediction Policy Optimization for Improved Visual Dynamics
Zirui Ge, Pengxiang Ding, Baohua Yin, Yemin Wang, Qishen Wang, Zhiyong Xie, Hengtao Li, Runze Suo, Wenxuan Song, Han Zhao, Shangke Lyu, Haoang Li, Ran Cheng, Cheng Chi, Hui-Bin Ge, Yaozhi Luo, Donglin Wang
Video action models are a promising foundation for Vision–Language–Action (VLA) because they can learn rich visual dynamics directly from video. However, likelihood-oriented training of diffusion predictors emphasizes globally plausible futures and does not guarantee precision-critical visual dynamics needed for manipulation, so small prediction errors can be amplified by downstream policies. We propose Dyn-VPP, a post-training framework that casts multi-step denoising as policy optimization and aligns predicted future latents with expert visual dynamics via verifiable terminal reward, without modifying any architecture. This enables explicit optimization of dynamics signals that are not captured by likelihood-only training. As a result, Dyn-VPP yields more accurate visual dynamics and improves downstream task execution. Experiments across diverse simulated and real-world manipulation settings show improved dynamics consistency and consistently higher task success.
video predictionVLAmanipulationpolicy optimization
DynVLA: Learning World Dynamics for Action Reasoning in Autonomous Driving
Shuyao Shang, Bing Zhan, Yunfei Yan, Yuqi Wang, Yingyan Li, Yasong An, Xiaoman Wang, Jierui Liu, Lu Hou, Lue Fan, Zhaoxiang Zhang, Tieniu Tan
We propose DynVLA, a driving VLA model that introduces a new CoT paradigm termed Dynamics CoT. DynVLA forecasts compact world dynamics before action generation, enabling more informed and physically grounded decision-making. To obtain compact dynamics representations, DynVLA introduces a Dynamics Tokenizer that compresses future evolution into a small set of dynamics tokens. Considering the rich environment dynamics in interaction-intensive driving scenarios, DynVLA decouples ego-centric and environment-centric dynamics, yielding more accurate world dynamics modeling. We then train DynVLA to generate dynamics tokens before actions through SFT and RFT, improving decision quality while maintaining latency-efficient inference. Compared to Textual CoT, which lacks fine-grained spatiotemporal understanding, and Visual CoT, which introduces substantial redundancy due to dense image prediction, Dynamics CoT captures the evolution of the world in a compact, interpretable, and efficient form. Extensive experiments on NAVSIM, Bench2Drive, and a large-scale in-house dataset demonstrate that DynVLA consistently outperforms Textual CoT and Visual CoT methods, validating the effectiveness and practical value of Dynamics CoT.
autonomous drivingchain-of-thoughtworld dynamicsvision-language-action
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 EARL: Towards a Unified Analysis-Guided Reinforcement Learning Framework for Egocentric Interaction Reasoning and Pixel Grounding
Yuejiao Su, Xinshen ZHANG, Zhen Ye, Lei Yao, Lap-Pui Chau, Yi Wang
A precise and comprehensive understanding of human-environment interactions in egocentric vision is essential for next-generation intelligent agents, such as assistive robotics.
While existing multimodal large language models (MLLMs) support unified reasoning from scene-level analysis to instance-specific grounding, their accuracy and generalization remain limited.
To this end, this paper introduces a novel Egocentric Analysis-guided RL-based method (EARL) that employs Group Relative Policy Optimization (GRPO) to enhance the interaction understanding of MLLMs in first-person vision.
Specifically, EARL adopts a two-stage parsing framework including coarse-grained interpretation and fine-grained response.
The first stage holistically interprets egocentric interactions and generates a structured textual description. The second stage produces the language answer and corresponding pixel-level grounding mask in response to the user query. To bridge the two stages, we extract a global interaction descriptor from the first stage and treat it as a semantic prior, which is then integrated via a novel Analysis-guided Feature Synthesizer (AFS) to support query-oriented reasoning.
Furthermore, to effectively guide policy optimization, we design a sophisticated, multi-faceted reward mechanism that incorporates format correctness, answer relevance, and grounding accuracy. Experimental results demonstrate that EARL achieves an impressive 65.48% cIoU on the Ego-IRGBench benchmark for pixel grounding, surpassing previous state-of-the-art RL-based methods by 8.37%.
Superior performance in out-of-distribution evaluations further validates EARL's generalization capability.
egocentric-visioninteraction-reasoningmllmpixel-grounding
OpenReview ↗
PDF ↗
Code ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 ECo-MoE: Embodiment-Conditioned Mixture of Experts Increases the Evolvability of Robots
Yibin Wang, Muhan Li, Zihan Guo, Sam Kriegman
In this paper, we introduce a model of evolution and learning in robots that co-optimizes a distribution of latent design vectors (genotypes) and a mixture of control experts (neural modules), which are gated by the latent coordinates of each decoded design (phenotype). This provides a scalable alternative to co-design algorithms that either train an individual policy for every robot, which is inefficient, or a monolithic universal controller for all robots, which results in overly conservative structures and behaviors. Our approach lies somewhere between these two extremes, preserving ancestral knowledge in a unified yet modular framework in which different body plans activate and deactivate different combinations of learned sensorimotor circuits for goal-directed behavior. This allows one part of the controller to be overhauled to better suit new species of designs as they emerge without disrupting the hard-earned knowledge contained within other expert modules. It also allows pretrained expert policies to be directly plugged into the mixture, which can steer evolution into otherwise unexplored areas of latent space containing desired morphological traits. We refer to this process as "evo by demo" and explore how it may be used to guide freeform evolution toward canonical structures defined by the pretrained model. Videos and code can be found at: https://eco-moe.github.io.
robot co-designevolutionary roboticsmorphologymixture of experts
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 EcoVLA: Environment-Aware Adaptive Pruning with Interleaved Inference Orchestration for Vision-Language-Action Models
Yuting Huang, Leilei Ding, Zhipeng Tang, Zenghuan Zhu, Jiajun Deng, Xinrui Lin, Shuo Liu, Haojie Ren, Jianmin Ji, Yanyong Zhang
While Vision-Language-Action (VLA) models hold promise in embodied intelligence, their large parameter counts lead to substantial inference latency that hinders real-time manipulation, motivating parameter sparsification. However, as the environment evolves during VLA execution, the optimal sparsity patterns change accordingly. Static pruning lacks the adaptability required for environment dynamics, whereas fixed-interval dynamic layer pruning suffers from coarse granularity and high retraining overheads. To bridge this gap, we propose **EcoVLA**, a training-free, plug-and-play adaptive pruning framework that supports orthogonal combination with existing VLA acceleration methods. EcoVLA comprises two components: **E**nvironment-aware **A**daptive **P**runing (**EAP**) and **I**nterleaved **I**nference **O**rchestration (**$I^2O$**). EAP is a lightweight adaptive channel pruning method that incorporates the temporal consistency of the physical environment to update sparsity patterns. $I^2O$ leverages the FLOPs bubbles inherent in VLA inference to schedule the pruning method in parallel, ensuring negligible impact on latency. Evaluated on diverse VLA models and benchmarks, EcoVLA delivers state-of-the-art performance, achieving up to 1.60$\times$ speedup with only a 0.4% drop in success rate, and further reaches 2.18$\times$ speedup with only a 0.5% degradation when combined with token pruning. We further validate the effectiveness of EcoVLA on real-world robots. Our code is available [here](https://github.com/Echo-hyt/Ecovla).
VLAmodel pruningefficient inferencereal-world robots
OpenReview ↗
PDF ↗
Code ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Efficient and Safe Molecular Assembly via Reinforcement Learning and Constraint Solving
Stefan Pranger, Bernhard Ramsauer, Oliver T. Hofmann, Bettina Könighofer
Scanning tunneling microscopy (STM) enables precise manipulation of individual atoms and molecules, offering a pathway to constructing nanoscale assemblies with rich quantum mechanical behavior. Despite its potential, STM-based fabrication remains limited by the inherent complexity of manipulation procedures and the extensive manual effort required. In this work, we take a substantial step toward autonomous manufacturing with STMs by introducing a novel AI-based planning framework for molecular assembly and a high-fidelity simulation environment. Our framework computes collision-free assembly plans that minimize the total distance traveled by molecules. Given an assignment of molecules to target positions, satisfiability solving is used to compute execution schedules in which each molecule has an empty corridor available when it is scheduled to move. Reinforcement learning (RL) agents then execute sequences of STM actions to manipulate molecules to their targets. We further introduce NanoAssemblyGym, a high-fidelity simulation environment for molecular manipulation built on the Gymnasium API, allowing seamless integration with existing RL libraries and workflows. Using NanoAssemblyGym, we demonstrate autonomous assembly of structures containing up to 420 molecules.
molecular-manipulationnanoscalerl-planningstm
OpenReview ↗
PDF ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 Efficient Skill Grounding via Code Refactoring with Small Language Models
Sera Choi, Wonje Choi, Saehun Chun, Daehee Lee, Jooyoung Kim, Chaeun Lee, Honguk Woo
Effective skill grounding is essential for deploying reusable skills in embodied agents, as even minor embodiment or environmental differences can render an entire skill incompatible.
This challenge is particularly pronounced in embodied settings, where agents must operate in dynamic, partially observable environments without access to large language models (LLMs).
In this setting, reliance on LLMs is impractical, while small language models (sLMs) remain insufficient for the effective skill grounding required for reliable long-horizon control.
We present RECENT, a refactoring-centric agent framework that enables efficient skill grounding with sLMs by decoupling skill semantics from embodiment- and environment-specific execution binding.
By representing skills as executable code, RECENT preserves the semantic intent encoded in a skill’s control structure while grounding it by modifying only execution bindings through localized refactoring, rather than regenerating code from scratch.
We evaluate RECENT across diverse skill grounding scenarios spanning multiple robot embodiments in dynamic environments, demonstrating robust long-horizon performance when deployed with an sLM.
Across all scenarios, RECENT achieves the best performance among sLM-based Code-as-Policies (CaP) methods and matches the task performance of LLM-based CaP.
skill groundingembodied agentscode-as-policiessmall language modelsmulti-embodiment
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 EgoTactile: Learning Grasp Pressure for Everyday Objects from Egocentric Video
Yuan Zeng, Yujia Shi, Tiao Tan, Xingting Li, Yaqi Qin, Zongqing Lu, Wenming Yang, Jing-Hao Xue, Qingmin Liao
Estimating full-hand grasp pressure from egocentric video is critical for immersive VR and robotic manipulation, yet dense tactile sensing often relies on intrusive hardware.
Existing vision-based methods predominantly rely on planar surfaces or fingertip contacts, failing to generalize to complex 3D object interactions.
Therefore, we introduce EgoTactile, a benchmark pairing egocentric video with full-hand pressure supervision for diverse everyday objects, incorporating a bare-hand transfer subset to enable generalization to natural scenarios.
Leveraging this benchmark, we first establish EgoPressureFormer as a discriminative baseline. Beyond this, to explicitly address the uncertainty in partial observations, we propose EgoPressureDiff, a conditional diffusion framework that adapts a large-scale pre-trained video diffusion backbone. By combining rich world knowledge priors with a Physically-Informed Feature Rectification layer to inject semantic constraints, our approach effectively hallucinates plausible contact patterns and resolves visual-physical ambiguities.
Extensive experiments demonstrate that our method achieves superior performance on the benchmark and robust transferability to in-the-wild scenarios. Our project page is at https://egotactile.github.io/.
grasp pressuretactile sensingegocentric visiondiffusion model
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 EMBGuard: Constructing Hazard-Aware Guardrails for Safe Planning in Embodied Agents
Dongwook Choi, Taeyoon Kwon, Bogyung Jeong, Minju Kim, Yeonjun Hwang, Hyojun Kim, Byungchul Kim, Young Kyun Jang, Jinyoung Yeo
MLLM-powered embodied agents deployed in real-world environments encounter physical hazards. However, existing approaches lack explicit mechanisms for identifying hazards and reasoning about action-conditioned risks, leading agents to either miss risky interactions or over-identify risks. To address this, we propose EMBGuard, the first MLLM-based safety guardrail for embodied agents designed to decouple physical risk reasoning from agent policy. By evaluating a (visual observation, action) pair, EMBGuard identifies hazardous configurations and provides natural language explanations of potential risks. Alongside EMBGuard, we contribute EMBHazard, a training dataset of 15.1K action-conditioned pairs, and EMBGuardTest, a benchmark of 329 manually curated real-world scenarios spanning seven physical risk categories. Through compositional variation of hazards and actions, we generate diverse risky and benign scenarios that agents may encounter during planning. Despite its compact size (2B, 4B), EMBGuard achieves performance competitive with proprietary MLLMs (e.g., GPT-5.1, Gemini-2.5-Pro) while significantly reducing the false-positive rates that hinder real-time deployment. We make the code, data, and models publicly available at https://github.com/dongwxxkchoi/EMBGuard.
embodied agentssafety guardrailphysical riskMLLM planning
OpenReview ↗
PDF ↗
Code ↗
Poster Session 6 (Hall A) · Thursday, July 9, 10:30-12:15 Embodied Interpretability: Linking Causal Understanding to Generalization in Vision-Language-Action Models
Hanxin Zhang, Mingshuo Xu, Abdulqader Dhafer, Shigang Yue, Hongbiao Dong, Zhou Daniel Hao
Vision–Language–Action (VLA) policies often fail under distribution shift, suggesting that decisions may depend on spurious visual correlations rather than task-relevant causes. We formulate visual–action attribution as an interventional estimation problem. Accordingly, we introduce the **Interventional Significance Score (ISS)**, an interventional masking procedure for estimating the causal influence of visual regions on action predictions, and the **Nuisance Mass Ratio (NMR)**, a scalar measure of attribution to task-irrelevant features. We analyze the statistical properties of ISS and show that it admits unbiased estimation, and we characterize conditions under which action prediction error provides a valid proxy for causal influence. Experiments across diverse manipulation tasks indicate that NMR predicts generalization behavior and that ISS yields more faithful explanations than existing interpretability methods. These results suggest that interventional attribution provides a simple diagnostic approach for identifying causal misalignment in embodied policies.
interpretabilitycausalityVLAgeneralization
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Embodied Task Planning via Graph-Informed Action Generation with Large Language Models
Xiang Li, Ning Yan, Masood S. Mortazavi
While Large Language Models (LLMs) have demonstrated strong zero-shot reasoning capabilities, their deployment as embodied agents still faces fundamental challenges in long-horizon planning. Unlike open-ended text generation, embodied agents must decompose high-level intents into actionable sub-goals while adhering to the constraints of a dynamic environment. Standard LLM planners frequently fail to maintain strategy coherence over extended horizons due to context window limitations or hallucinate state transitions that violate environment constraints. We propose GiG, a planning framework that structures embodied agents’ memory using a Graph-in-Graph architecture. Our approach employs a Graph Neural Network (GNN) to encode environmental states into embeddings, organizing these embeddings into action-connected execution trace graphs within an experience memory bank. GiG enables retrieval of structurally-similar priors, allowing agents to ground current decisions in relevant past structural patterns. Furthermore, we introduce a bounded lookahead module that leverages symbolic transition logic to enhance the agent’s planning capabilities through grounded action projections. We evaluate our framework on three embodied planning benchmarks—Robotouille Synchronous, Robotouille Asynchronous, and ALFWorld. Our method outperforms state-of-the-art baselines, achieving Pass@1 performance gains of up to 22% on Robotouille Synchronous, 37% on Asynchronous, and 15% on ALFWorld while maintaining comparable or lower computational cost.
LLM planningembodied agentsALFWorldgraph neural networklong-horizon planning
OpenReview ↗
PDF ↗
Poster Session 7 (Hall A) · Thursday, July 9, 2:30-4:15 Embodied-DETR: End-to-End Temporal 3D Object Detection in Egocentric Views
Ziheng Ding, Xiaze Zhang, Yuejie Zhang, lifeng chen, Rui Feng
Embodied 3D object detection is a fundamental perceptual capability for embodied agents, in which observations are partial, heavily occluded, and sequential, requiring modeling of temporal continuity.
However, existing benchmarks and methods are primarily designed for fully reconstructed global scenes and fail to capture temporal observation context and instance evolution in first-person perception.
We introduce **Embodied-Det**, a new benchmark for embodied 3D object detection that evaluates detection accuracy, temporal stability, and consistency under egocentric sequential views.
Building on this benchmark, we propose **Embodied-DETR**, an end-to-end temporal detection framework that models scene-level context and instance-level consistency through two complementary temporal modules, *Scene-aware Feature Aggregation* and *Instance-aware Query Embedding*.
Experiments on Embodied-Det show that existing methods suffer substantial performance degradation in egocentric temporal settings, while Embodied-DETR achieves superior accuracy and temporal consistency, demonstrating the effectiveness of temporal modeling for embodied 3D perception.
Codes are available at https://github.com/UniPerceptor/UniPerceptor.
3D object detectionegocentric perceptiontemporal modelingbenchmark
OpenReview ↗
PDF ↗
Code ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 EnerGS: Energy-Based Gaussian Splatting under Partial Geometric Priors
Rui Song, Tianhui Cai, Markus Gross, Yun Zhang, Walter Zimmer, Zhiyu Huang, Olaf Wysocki, Jiaqi Ma
3D Gaussian Splatting (3DGS) has been widely adopted for scene reconstruction, where training inherently constitutes a highly coupled and non-convex optimization problem. Recent works commonly incorporate geometric priors, such as LiDAR measurements, either for initialization or as training constraints, with the goal of improving photometric reconstruction quality. However, in large-scale outdoor scenarios, such geometric supervision is often spatially incomplete and uneven, which limits its effectiveness as a reliable prior and can even be detrimental to the final reconstruction. To address this challenge, we model partially observable geometry as a continuous energy field induced by geometric evidence and propose EnerGS. Rather than enforcing geometry as a hard constraint, EnerGS provides a soft geometric guidance for the optimization of Gaussian primitives, allowing geometric information to steer the optimization process without directly restricting the solution space. Extensive experiments on large-scale outdoor scenes demonstrate that, under both sparse multi-view and monocular settings, EnerGS consistently improves photometric quality and geometric stability, while effectively mitigating overfitting during 3DGS training. The codebase is publicly available at: https://github.com/ucla-mobility/EnerGS.
gaussian-splattingscene-reconstructionautonomous-driving3d-reconstruction
Energy-based Compositional Diffusion Planning
Tao Sun, Utkarsh Aashu Mishra, Jiaxin Lu, Danfei Xu, Iro Armeni
Compositional diffusion planners aim to solve long-horizon robotic tasks using short training trajectories. Yet, current approaches often rely on the heuristic stitching of local predictions. We show that the resulting stitched update is generally a non-conservative field} that does not mathematically correspond to any valid global trajectory log-density function. We propose **Energy-based Compositional Diffuser** (ECD), a framework that formulates the global trajectory as the minimizer of the sum of local bridge potentials. This energy-based perspective defines a conservative correction field and contains a boundary reaction term that heuristic stitching omits. To enable efficient inference, we further introduce a Markov-based score approximation that computes the reaction term via a single block-tridiagonal solve, maintaining time complexity linear in the planning horizon. Empirically, ECD achieves state-of-the-art success rates on a range of OGBench stitching tasks, while nearly matching the inference speed of heuristic stitching methods. Code is available at https://github.com/GradientSpaces/ECD.
diffusion planningrobot planningcompositional planninglong-horizon
EnsembleVLA: Ensemble Learning for Vision-Language Action Models
Mingchen Song, Xiang Deng, Jie Wei, Dongmei Jiang, Liqiang Nie, Weili Guan
Diverse Vision-language-action (VLA) models have been proposed and demonstrated remarkable capabilities in robotic manipulation. However, how to effectively ensemble VLAs to further enhance performance remains largely unexplored, as conventional ensemble techniques designed for discriminative tasks cannot be directly applied to generative action policies with high-dimensional, multimodal distributions. To address this challenge, we propose EnsembleVLA, an energy-based framework that enables principled ensemble of diverse VLA models. We establish a unified theoretical framework showing that both diffusion-based and flow-based VLA models can be formulated as energy-based models, where additive energy combination naturally induces policy composition at the distribution level. This theoretical foundation enables multiple pre-trained policies to be seamlessly aggregated into a stronger ensemble policy. Building upon this compositional framework, EnsembleVLA further incorporates learnable composition weights for dynamic policy balancing, coupled with a confidence-aware gating mechanism that adaptively modulates bounded residual corrections, collectively ensuring stable and robust task execution. Extensive experiments demonstrate that EnsembleVLA achieves competitive performance across various tasks in both simulated and real-world environments.
VLAensemble learningenergy-based modelsrobotic manipulation
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 EPS3D: End-to-End Feed-Forward 3D Panoptic Segmentation
Runsong Zhu, Jiaxin Guo, Xiaoyang Guo, Zhengzhe Liu, Ka-Hei Hui, Wei Yin, Kai Chen, Wei Chen, Weiqiang Ren, Yunhui Liu, Pheng-Ann Heng, Chi-Wing Fu
This paper introduces EPS3D, a new end-to-end feed-forward framework for open-vocabulary 3D panoptic segmentation. Unlike existing methods relying on additional preprocessing, we design an end-to-end architecture, with a distillation-based training strategy on diverse 3D scenes to predict 3D-aware semantic and instance features from multi-view images, improving 3D consistency and avoiding error accumulation. We further propose a mutual enhancement module to enforce inherent semantic-instance consistency. By aligning semantics within instances (Ins2Sem) and refining instance features with semantic guidance (Sem2Ins), we achieve more coherent 3D scene understanding. Ultimately, EPS3D outperforms SOTA baselines on two benchmarks (e.g., +13% mIoU for semantics on Replica) with high efficiency (e.g., 1s per scene), supporting tasks like robotic manipulation and 3D scene editing.
3d-segmentationopen-vocabularypanoptic-segmentationperception
OpenReview ↗
PDF ↗
Poster Session 6 (Hall A) · Thursday, July 9, 10:30-12:15 Equivariant Neural Networks for General Linear Symmetries on Lie Algebras
Chankyo Kim, Sicheng Zhao, Minghan Zhu, Tzu-Yuan Lin, Maani Ghaffari
Many scientific and geometric problems exhibit general linear symmetries, yet most equivariant neural networks are built for compact groups or simple vector features, limiting their reuse on matrix-valued data such as covariances, inertias, or shape tensors.
We introduce \textbf{Reductive Lie Neurons (ReLNs)}, an exactly $\mathrm{GL}(n)$-equivariant architecture that natively supports matrix-valued and Lie-algebraic features. ReLNs resolve a central stability issue for reductive Lie algebras by introducing a non-degenerate adjoint (conjugation)-invariant bilinear form, enabling principled nonlinear interactions and invariant feature construction in a single architecture that \textit{transfers across subgroups without redesign}. We demonstrate ReLNs on algebraic tasks with $\mathfrak{sl}(3)$ and $\mathfrak{sp}(4)$ symmetries, Lorentz-equivariant particle physics, uncertainty-aware drone state estimation via joint velocity--covariance processing, learning from 3D Gaussian-splat representations, and EMLP double-pendulum benchmark spanning multiple symmetry groups. ReLNs consistently match or outperform strong equivariant and self-supervised baselines while using substantially fewer parameters and compute, improving the accuracy–efficiency trade-off and providing a practical, reusable backbone for learning with broad linear symmetries.
equivariant-networkslie-algebrasgeometric-deep-learningstate-estimation
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 6 (Hall A) · Thursday, July 9, 10:30-12:15 Escaping the Diversity Trap in Robotic Manipulation via Anchor-Centric Adaptation
Yanzhe Chen, Kevin Yuchen Ma, Qi Lv, Yiqi Lin, Zechen Bai, Chen Gao, Mike Zheng Shou
While Vision-Language-Action (VLA) models offer broad general capabilities, deploying them on specific hardware requires real-world adaptation to bridge the embodiment gap. Since robot demonstrations are costly, this adaptation must often occur under a strict data budget. In this work, we identify a critical **diversity trap**: the standard heuristic of ``maximizing coverage" by collecting diverse, single-shot demonstrations can be self-defeating due to non-vanishing estimation noise. We formalize this phenomenon as a **Coverage--Density Trade-off**. By decomposing the policy error into estimation (density) and extrapolation (coverage) terms, we characterize an interior optimal allocation of unique conditions for a fixed budget. Guided by this analysis, we propose **Anchor-Centric Adaptation (ACA)**, a two-stage framework that first stabilizes a policy skeleton through repeated demonstrations at core anchors, then selectively expands coverage to high-risk boundaries via teacher-forced error mining and constrained residual updates. Real-robot experiments validate our trade-off framework and demonstrate that ACA significantly improves task reliability and success rates over standard diverse sampling strategies under the same budget.
VLA adaptationrobot manipulationdata-efficient learningreal-robot
OpenReview ↗
PDF ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 EvoCF: Multi-Agent Collaboration via Agentic Memory-Driven Evolutionary Counterfactual Planning
Haotian Chi, Zeyu Feng, Xingrui Yu, Linbo Luo, Yew-Soon Ong, Ivor Tsang, Hechang Chen, Yi Chang, Haiyan Yin
Planning collaboration strategies for multi-agent embodied systems remains a core challenge for LLM-based planners, which often fail to capture the physical and coordination constraints of realworld environments. To address this, we present EvoCF, an agentic memory-driven evolutionary counterfactual planning framework for discovering improved multi-agent collaboration strategies through counterfactual plan generation and evaluation. First, we propose a symbolic constraint inductor that induces reusable symbolic constraints from failures, forming an evolving rule library. Then, we propose an evolutionary counterfactual plan generator that systematically explores semantically consistent plan variants through rule-conditioned mutations, enabling robust collaboration strategies beyond short-sighted one-shot LLM plans. Finally, we design an agentic memory-grounded evaluator that ranks candidate plans using retrieval-augmented evidence, producing interpretable, constraint-aware selections. Across multi-agent embodied simulation benchmarks, EvoCF consistently discovers more robust and executable plans compared to baseline approaches. Our results demonstrate that grounding multi-agent planning in agentic memory and counterfactual reasoning significantly enhances both effectiveness and robustness.
multi-agentplanningllm-agentsembodied-systems
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Factored Latent Action World Models
Zizhao Wang, Chang Shi, Jiaheng Hu, Kevin Rohling, Roberto Martín-Martín, Amy Zhang, Peter Stone
Learning latent actions from action-free video has emerged as a powerful paradigm for scaling up controllable world model learning. Latent actions provide a natural interface for users to iteratively generate and manipulate videos. However, most existing approaches rely on monolithic inverse and forward dynamics models that learn a single latent action to control the entire scene, and therefore struggle in complex environments where multiple entities act simultaneously. This paper introduces Factored Latent Action Model (FLAM), a factored dynamics framework that decomposes the scene into independent factors, each inferring its own latent action and predicting its own next-step factor value. This factorized structure enables more accurate modeling of complex multi-entity dynamics and improves video generation quality in action-free video settings compared to monolithic models. Based on experiments on both simulation and real-world multi-entity datasets, we find that FLAM outperforms prior work in prediction accuracy and representation quality, and facilitates downstream policy learning, demonstrating the benefits of factorized latent action models.
world-modelslatent-actionsvideo-predictionmulti-agent
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 Few-Shot Design Optimization by Exploiting Auxiliary Information
Arjun Mani, Carl Vondrick, Richard Zemel
Many real-world design problems involve optimizing an expensive black-box function $f(x)$, for which Bayesian Optimization is a sample-efficient framework. However, while the basic black-box setting returns a scalar reward, real-world experiments often generate a wealth of useful information. We introduce a new setting where an experiment generates high-dimensional auxiliary information $h(x)$ along with $f(x)$; moreover, a history of relevant, previously-solved tasks is available for accelerating optimization. We develop a novel method based on a neural model which predicts $f(x)$ for unseen designs given a few-shot context containing observations of $h(x)$. We evaluate our method on two challenging domains, robotic hardware design and hyperparameter tuning. On both domains, our method achieves improved few-shot prediction and faster design optimization, outperforming several multi-task optimization methods.
bayesian-optimizationdesign-optimizationfew-shot-learningblack-box-optimization
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 Fisher-Preserving Guidance: Training-Free Manifold Constraints for Safe Diffusion Control
Hao Ren, Zetong Bi, Yiming Zeng, Le Zheng, Zhi Li, Zhaoliang Wan, Lu Qi, Hui Cheng
Diffusion models are effective for waypoint prediction in visual navigation, but standard sampling and test time guidance can produce unreliable or inefficient trajectories when updates drift off the training manifold. We propose Fisher Preserving Guidance with Outer Product Span Projection, a training-free inference method that avoids large Fisher drift associated with off-distribution actions while optimizing a task objective. Our method computes the Fisher-preserving update via a low-rank Jacobian factorization, requiring only a single backward pass per step and enabling real-time use. We further introduce Truncated Fisher Denoising Sensitivity as an uncertainty signal and use it for robust multi-sample action blending. Experiments on toy and realistic navigation benchmarks, including Maze2D with TSDF-based guidance, PushT with official Diffusion Policy weights, and visual navigation in simulation and on real robots, demonstrate consistent improvements in performance over strong diffusion-policy baselines without additional training.
visual navigationdiffusion policyreal robotsuncertainty estimation
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 FlatLab: A Unified Methodology Framework and Simulation-Based Benchmark for Robotic Manipulation of Flat Objects
Xingyu Zhu, Wenshuo Han, Zhouyu Wang, Yuran Wang, Ruihai Wu, Hao Dong, Fan Tang, Hechang Chen, Hyung Jin Chang, Yixing Gao
Robotic manipulation of flat objects is challenging due to the ungraspable configurations and strong variations in object geometry and material. Existing methods rely on heuristic pre-manipulation and are often evaluated in closed settings with limited generalization. We propose a unified framework that decouples the manipulation into a strategy generator and an action execution module. The strategy generator predicts appropriate manipulation strategies from object point clouds by learning strategy-centric, object-invariant representations via simulated data transformation and contrastive learning. Conditioned on the predicted strategy, the execution module decomposes long-horizon manipulation into reusable action primitives and dynamically composes them to generate stable trajectories. To enable systematic evaluation, we introduce FlatLab, a comprehensive simulation benchmark for robotic flat object manipulation. FlatLab provides high-fidelity physical simulation of diverse rigid and deformable flat objects, automated multi-modal data collection, and standardized task definitions and evaluation protocols. Experiments conducted in FlatLab demonstrate that our approach generalizes effectively to unseen objects and categories, outperforming existing baselines. The project page and the code are provided at \url{https://flatlab-web.github.io/}.
robotic manipulationflat object graspingsimulation benchmarkcontrastive learning
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Flow Equivariant World Models: Structured Memory for Dynamic Environments
Hansen Lillemark, Benhao Huang, Fangneng Zhan, Yilun Du, T. Anderson Keller
Embodied systems experience the world as 'a symphony of flows': a combination of many continuous streams of sensory input coupled to self-motion, interwoven with the dynamics of external objects. These sensory streams and the underlying dynamics of the world obey smooth, time-parameterized symmetries which existing world models ignore. Without a memory that respects this structure, partial observability presents a major obstacle to existing methods: each observation reveals only a fraction of the world, while unobserved regions continue to evolve. In this work, we introduce Flow Equivariant World Modeling, a framework that leverages time-parameterized symmetries within a latent memory for stable and accurate dynamics prediction over long horizons. The latent memory shifts and transforms equivariantly with self-motion and inferred external object motion, keeping information about out-of-view regions aligned as time progresses. We demonstrate the advantage of this framework over state-of-the-art diffusion, memory-augmented, and recurrent world model architectures on 2D and 3D partially observed video world modeling benchmarks.
More broadly, our results suggest that predictive representations become more powerful when they are organized in line with the temporal and dynamical structure of the world they model. Project page: https://flowequivariantworldmodels.github.io/
world-modelsequivariancespatial-memoryvideo-generation
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 6 (Hall A) · Thursday, July 9, 10:30-12:15 FOCA: Future-Oriented Conditioning for Data-Efficient Vision-Language-Action Adaptation
Minh Duc Nguyen, Nghiem Tuong Diep, Nguyen Gia Binh, Trong-Bao Ho, Doanh Le Thien, Quang Tan Nguyen, Thien-Loc Ha, Tran Van Nhiem, Bao Thach, Tran Xuan Nhat, Tuan Anh Tran, Artur Habuda, Philip Lund Møller, Tran Nguyen Le, Daniel Sonntag, Mathias Niepert, Khoa D Doan, Vu N. Duong, Hung Ngo, Minh Nhat VU, Duy Minh Ho Nguyen, An Thai Le, Vien Anh Ngo
Vision–Language–Action (VLA) models enable general-purpose robotic control via large-scale multimodal pretraining, yet their effectiveness under few-shot imitation learning remains limited. We conduct a systematic stress test of state-of-the-art VLA models and show that performance degrades sharply as demonstrations are reduced, revealing a key weakness of existing adaptation strategies. To address this, we introduce FOCA, a future-oriented conditioning framework for data-efficient VLA adaptation. FOCA combines explicit prediction of task-grounded future interaction embeddings with implicit alignment to future goal observations, enabling long-horizon reasoning in latent space without pixel-level prediction. This formulation naturally supports action-free co-training with synthetic videos from video world models and can be interpreted as learning a future-conditioned value-like representation. Extensive experiments demonstrate FOCA achieves 95.7\% success with 20 demonstrations on LIBERO, improves 7–12\% on RoboCasa, and delivers up to 26\% absolute gains on real robots, establishing a new state of the art in few-shot VLA adaptation.
VLAfew-shot imitation learningLIBERORoboCasareal robots
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 FocalPolicy: Frequency-Optimized Chunking and Locally Anchored Flow Matching for Coherent Visuomotor Policy
Qian He, Zhenshuo Yang, Wenqi Liang, Chunhui Hao, Nicu Sebe, Jiandong Tian
Visuomotor policies aim to learn complex manipulation tasks from expert demonstrations.
However, generating smooth and coherent trajectories remains challenging, as it requires balancing proximal precision with distal foresight.
Existing approaches typically focus on optimizing intra-chunk action distributions, often neglecting the inter-chunk coherence.
Consequently, inter-chunk discontinuities significantly impede the learning of coherent long-horizon actions.
To overcome this limitation and achieve a synergetic balance between precision and foresight, we propose FocalPolicy, a foresight-aware visuomotor policy that combines Frequency-Optimized Chunking with Locally Anchored flow matching.
We introduce a foresight composite objective that supervises time-domain alignment within the proximal actions while regularizing frequency-domain structure over multiple future action chunks to improve cross-chunk coherence.
To efficiently learn complex action distributions, we design locally anchored sampling to enhance target signal propagation efficiency during consistency flow matching training.
Extensive experiments demonstrate that FocalPolicy outperforms existing approaches and confirm the generalizability of our modules to other baselines.
Project website: https://focalpolicy.github.io/
visuomotor policyimitation learningflow matchingmanipulation
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Focus-Then-Contact: Speeding Up Robotic Contact-Rich Task Learning with Affordance-Guided Real-World Residual Reinforcement Learning
Guanren Qiao, Ruixiang Ouyang, Sheng Xu, Ruixing Jin, Yueci Deng, Yunxin Tai, Kui Jia, Guiliang Liu
Real-World Reinforcement Learning (RL) has shown significant potential in robotic manipulation tasks. However, many methods still require substantial human-in-the-loop involvement to complete contact-rich tasks, especially when there are disruptions such as visual backgrounds or positional changes. To address this, we propose the Focus Then Contact (FTC), a lightweight and low-cost method to accelerate the convergence of human-in-the-loop real-world RL for contact-rich tasks. FTC leverages residual RL to provide base actions, helping the system quickly reach the target regions and improve sample efficiency. Additionally, FTC integrates an affordance-guided reward that drives the real-world RL system to quickly focus on key regions of interest, making it possible for the robotic arm to continuously engage with these goal areas through force-control feedback. At the same time, we optimize the human-in-the-loop implementation to prevent conflicts with RL over control of the robotic arm. We demonstrate the effectiveness of FTC on 6 contact-rich tasks, where it outperforms baseline methods in achieving high success rates and speeds up robotic contact-rich task learning under a real-world RL setting.
contact-rich manipulationresidual RLaffordancereal-world RLhuman-in-the-loop
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Fourier Features Let Agents Learn High Precision Policies with Imitation Learning
Balázs Gyenes, Emiliyan Gospodinov, Jan Frieling, Enrico Krohmer, Nicolas Schreiber, Xiaogang Jia, Niklas Freymuth, Gerhard Neumann
High-precision robotic manipulation requires fine-grained spatial reasoning that is often difficult to achieve with RGB-only policies due to depth ambiguity and perspective scale issues. Policies that leverage 3D information directly, such as those based on point clouds, offer a stronger geometric prior over purely image-based ones, yet their performance remains highly task-dependent. We hypothesize that this discrepancy may be due to the spectral bias of neural networks towards learning low frequency functions, which especially affects architectures conditioned on slow-moving Cartesian features. We thus propose to map point clouds from Cartesian space into high-dimensional Fourier space, effectively equipping the point cloud encoder with direct access to high-frequency features. We experimentally validate the use of Fourier features on challenging manipulation tasks from the RoboCasa and ManiSkill3 benchmarks and on a real robot setup. Despite their simplicity, we find that Fourier features provide significant benefits across diverse encoder architectures and benchmarks and are robust across hyperparameters. Our results indicate that Fourier features let policies leverage geometric details more effectively than Cartesian features, showing their potential as a general-purpose tool for point cloud-based imitation learning. We provide source code and videos on our project page: https://fourier-il.github.io/fourier-il.
imitation learningpoint cloudshigh-precision manipulationreal robot
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 From Abstraction to Instantiation: Learning Behavioral Representation for Vision-Language-Action Model
Bing Hu, Zaijing Li, Rui Shao, Junda Chen, April Hua Liu, Wei-Shi Zheng, Liqiang Nie
Vision-Language-Action (VLA) models often suffer from performance degradation under distribution shifts, as they struggle to learn generalized behavior representations across varying environments. While existing approaches attempt to construct behavior representations through action-centric latent variables, they are often limited by short-horizon temporal fragmentation and static execution-alignment, leading to inconsistent behaviors in complex scenarios. To address these limitations, we propose \textbf{BehaviorVLA}, a framework that facilitates robust manipulation through the learning of a temporally coherent behavioral representations. Our approach features two symmetric components: (1) the \textbf{Visuomotor Behavior Encoder (VBE)}, which utilizes a causal Mamba-based architecture to aggregate long-horizon trajectory information into a unified behavior representation; and (2) the \textbf{Phase-conditioned Behavior Decoder (PBD)}, which decodes this representation into precise actions by dynamically aligning task-level priors with real-time execution progress. Experiments on RoboTwin 2.0, LIBERO, and CALVIN demonstrate state-of-the-art success rates of 58\%, 98\%, and 4.36 (Avg.Len), respectively. Notably, in real-world sim-to-real transfer, BehaviorVLA matches the performance of OpenVLA-OFT using only 50\% of the demonstration data, showcasing its superior data efficiency and generalization.
VLArobotic manipulationsim2realRoboTwinCALVIN
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation
Yajie Li, Bozhou Zhang, Chun Gu, Zipei Ma, Jiahui Zhang, Jiankang Deng, Xiatian Zhu, Li Zhang
Video generation models offer a promising imagination mechanism for robot manipulation by predicting long-horizon future observations, but effectively exploiting these imagined futures for action execution remains challenging. Existing approaches either condition policies on predicted frames or directly decode generated videos into actions, both suffering from a mismatch between visual realism and control relevance. As a result, predicted observations emphasize perceptual fidelity rather than action-centric causes of state transitions, leading to indirect and unstable control.
To address this gap, we propose MoLA (Mixture of Latent Actions), a control-oriented interface that transforms imagined future videos into executable representations. Instead of passing predicted frames directly to the policy, MoLA leverages multiple pretrained inverse dynamics models to infer a mixture of latent actions implied by generated visual transitions. These modality-aware inverse dynamics models capture complementary semantic, depth, and flow cues, providing a structured and physically grounded action representation that bridges video imagination and policy execution.
We evaluate our approach on simulated benchmarks (LIBERO, CALVIN, and LIBERO-Plus) and real-world robot manipulation tasks, achieving consistent gains in task success, temporal consistency, and generalization.
robot manipulationlatent actionsvideo generationinverse dynamics
From Noise to Control: Parameterized Diffusion Policies
Renhao Zhang, Haotian Fu, Mingxi Jia, George Konidaris, Yilun Du, Bruno Castro da Silva
We propose Parameterized Diffusion Policy (PDP), a framework that learns a diffusion policy parameterized in a smooth continuous space. By structuring a latent manifold such that distances between latents' values reflect the semantic similarity of physical trajectories, we transform diffusion from a mechanism of stochastic diversity into a precise tool for behavior steering. Our approach also enables smooth interpolation between known strategies and efficient generalization to novel constraints without the need to update policy weights. We demonstrate that PDP significantly improves adaptation performance on complex multimodal benchmarks in both simulation and real-robot hardware compared to regular diffusion policy, particularly in scenarios requiring the discovery of novel behaviors.
diffusion policyreal-robotmultimodal behavior learning
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 From Noise to Intent: Anchoring Generative VLA Policies with Residual Bridges
Yiming Zhong, Yaoyu He, Zemin Yang, Pengfei Tian, Yifan Huang, Qingqiu Huang, Xinge Zhu, Yuexin Ma
Bridging high-level semantic understanding with low-level physical control remains a persistent challenge in embodied intelligence, stemming from the fundamental spatiotemporal scale mismatch between cognition and action. Existing generative VLA policies typically adopt a "Generation-from-Noise" paradigm, which disregards this disparity, leading to representation inefficiency and weak condition alignment during optimization. In this work, we propose ResVLA, an architecture that shifts the paradigm to "Refinement-from-Intent." Recognizing that robotic motion naturally decomposes into global intent and local dynamics, ResVLA utilizes spectral analysis to decouple control into a deterministic low-frequency anchor and a stochastic high-frequency residual. By anchoring the generative process on the predicted intent, our model focuses strictly on refining local dynamics via a residual diffusion bridge. Extensive simulation experiments show that ResVLA achieves competitive performance, strong robustness to language and robot embodiment perturbations, and faster convergence than standard generative baselines. ResVLA also demonstrates strong performance in real-world robot experiments.
VLAresidual diffusionrobotic action generationreal-world robot
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 From Perception to Planning: Evolving Ego-Centric Task-Oriented Spatiotemporal Reasoning via Curriculum Learning
Xiaoda Yang, Yuxiang Liu, Shenzhou Gao, Can Wang, Jingyang Xue, Lixin Yang, Yao Mu, Tao Jin, Zhimeng Zhang, Shuicheng YAN, Zhou Zhao
Modern vision-language models achieve strong performance in static perception, but remain limited in the complex spatiotemporal reasoning required for embodied, egocentric tasks. A major source of failure is their reliance on temporal priors learned from passive video data, which often leads to spatiotemporal hallucinations and poor generalization in dynamic environments. To address this, we present EgoTSR, a curriculum-based framework for learning task-oriented spatiotemporal reasoning. EgoTSR is built on the premise that embodied reasoning should evolve from explicit spatial understanding to internalized task-state assessment and finally to long-horizon planning. To support this paradigm, we construct EgoTSR-Data, a large-scale dataset comprising 46 million samples organized into three stages: Chain-of-Thought supervision for explicit reasoning, weakly supervised tagging for internalizing reasoning, and long-horizon sequences composed of orthogonal atomic subtasks. We further introduce a Reasoning-Enhanced Task Decomposition mechanism that explicitly models causal dependencies between actions and objects, enabling controllable planning from high-level task descriptions. To rigorously validate our approach, we establish a Dual-Level Evaluation Framework that covers both atomic spatial perception and logical planning. Extensive experiments demonstrate that EgoTSR effectively eliminates chronological biases, achieving 92.4% accuracy on long-horizon logical reasoning tasks while maintaining 88.2% fine-grained perceptual precision, significantly outperforming existing open-source and closed-source state-of-the-art models.
egocentric-visionspatiotemporal-reasoningvlmcurriculum-learning
OpenReview ↗
PDF ↗
Poster Session 1 (Hall A) · Tuesday, July 7, 10:30-12:15 From Pixels to Tokens: A Systematic Study of Latent Action Supervision for Vision-Language-Action Models
Yihan Lin, Haoyang Li, Yang Li, Haitao Shen, Yihan Zhao, Chao Shao, Jing Zhang
Latent actions serve as an intermediate representation that enables consistent modeling of vision-language-action (VLA) models across heterogeneous datasets. However, approaches to supervising VLAs with latent actions are fragmented and lack a systematic comparison. This work structures the study of latent action supervision from two perspectives: (i) regularizing the trajectory via image-based latent actions, and (ii) unifying the target space with action-based latent actions. Under a unified VLA baseline, we instantiate and compare four representative integration strategies. Our results reveal a formulation-task correspondence: image-based latent actions benefit long-horizon reasoning, whereas action-based latent actions excel at complex motor coordination. Furthermore, we find that directly supervising the VLM with discrete latent action tokens yields the most effective performance. Finally, our experiments offer initial insights into the benefits of latent action supervision in mixed-data, suggesting a promising direction for VLA training.
VLAlatent actionsrobot manipulationbenchmark study
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuning
Zhanyi Sun, Shuran Song
We introduce Distribution Contractive Reinforcement Learning (DICE-RL), a framework that uses reinforcement learning (RL) as a "distribution contraction" operator to refine pretrained generative robot policies. DICE-RL turns a pretrained behavior prior into a high-performing "pro" policy by amplifying high-success behaviors from online feedback. We pretrain a diffusion- or flow-based policy for broad behavioral coverage, then finetune it with a stable, sample-efficient residual off-policy RL framework that combines selective behavior regularization with value-guided action selection. Extensive experiments and analyses show that DICE-RL reliably improves performance with strong stability and sample efficiency. It enables mastery of complex long-horizon manipulation skills directly from high-dimensional pixel inputs, both in simulation and on a real robot. Project website: [dice.rl.2026](https://zhanyisun.github.io/dice.rl.2026/).
dexterous manipulationdiffusion policyrl finetuningreal robotlong-horizon
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Functional Cache Grafting: Robust and Rapid Code-Policy Synthesis for Embodied Agents
Saehun Chun, Wonje Choi, Sera Choi, Sanghyun Ahn, Honguk Woo
Code-writing large language models (CodeLLMs) generate executable code policies for embodied agents by translating natural language goals and environmental constraints into structured control programs. However, policy generation in open-domain embodied environments suffers from two fundamental limitations:
(i) delayed decoding caused by repetitive prefill computation over long prompts, and
(ii) limited robustness due to fully generative decoding, which often produces API mismatches, missing safety guards, and unstable control logic. To address these limitations, we present FCGraft, a Functional Cache Grafting framework.
FCGraft maintains a library of function-level validated code skeletons and their associated prompt-level Transformer key–value (KV) caches, and synthesizes new policies by retrieving relevant functions and grafting their KV caches when a new task is provided. Given retrieved function caches, FCGraft performs cache grafting via stitching, which composes cached function segments into a composite policy, and patching, which locally adapts only the necessary code regions to satisfy task-specific parameters and constraints with minimal additional decoding. By eliminating redundant prefill computation, this approach reduces generation latency, while reusing validated control structures improves robustness over prompt-level caching methods RAGCache, achieving $18.31\$% higher task success rate and $2.3\times$ faster policy synthesis.
code-as-policyllmembodied-agentsinference-efficiency
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 GAE: Unleashing Physical Potential of VLM with Generalizable Action Expert
Mingyu Liu, Zheng Huang, Xiaoyi Lin, Muzhi Zhu, Canyu Zhao, Yating Wang, Haoyi Zhu, Hao Chen, Chunhua Shen
Vision-language models demonstrate strong reasoning and planning abilities, yet grounding these predictions into precise robot actions remains a central challenge. Existing Vision-Language-Action methods typically entangle reasoning and action generation, leading to limited generalization and costly adaptation. We propose to learn a \textbf{G}eneralizable \textbf{A}ction \textbf{E}xpert (\textbf{GAE}), a task-agnostic model that converts sparse geometric plans into dense robot actions. Our approach introduces a sparse geometric interface: the VLM predicts sparse 3D waypoints representing high-level intention, while GAE maps these waypoints together with real-time point cloud observations to continuous action trajectories. GAE is pretrained on a large-scale pointcloud–trajectory dataset comprising \textbf{150k} trajectories from both simulation and real-world robots. To further improve efficiency and generalization, we introduce an \textbf{Action Pre-training, Pointcloud Fine-tuning (APPF)} scheme that decouples learning action dynamics from geometry grounding. After pretraining, GAE is frozen and reused across downstream tasks, requiring only lightweight fine-tuning of the VLM to produce the sparse interface. Extensive experiments show that our method achieves strong performance and generalization across diverse visual domains, camera viewpoints, and natural language instructions.
vision-language-actionwaypoint predictionpoint cloudgeneralization
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 General Covariant Action Modeling: Constructing Generalized Manifolds via Spatio-Temporal Decoupling
Huaihai Lyu, Chaofan Chen, Mingyu Cao, Yuheng Ji, Changsheng Xu
Achieving robust generalization from limited data is a central challenge in embodied intelligence. Prevailing methods fail by regressing absolute coordinates, which violates the principle of general covariance. Theoretically, this conflates the intrinsic task geometry with rigid execution patterns, binding policies to specific motion styles and fixed speeds. To resolve this, we propose the Generalized Action Manifold (GAM) framework that enforces general covariance through structural disentanglement. Specifically, GAM constructs the manifold by enforcing invariance across two orthogonal dimensions: (1) Temporal Invariance, utilizing an Arc-Length Parameterizer to orthogonalize the spatial path geometry from temporal dynamics, ensuring robustness to velocity variations; (2) Geometric Invariance, where a Schema-Affine-Factorization mechanism maps trajectories to canonical “world lines” in the Lie-algebraic tangent space. This distinguishes invariant topological schemas from affine modulations, ensuring spatial generalizability. By integrating GAM within a structured Vision-Language-Action (VLA) architecture, we expand sparse training data into a continuous, valid action manifold. Empirical results demonstrate that GAM enables superior transfer and robustness capabilities, significantly outperforming geometry-agnostic baselines.
vlaaction manifoldgeneralizationrepresentation learning
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Generalizable and Actionable Parts Pose Estimation with Symmetry Annotation-Free Learning Strategy
wenxiao chen, Xueyu Yuan, Liu Liu, Di Wu, Dan Guo
Urgently needed generalizable robot object interaction and manipulation requires high-quality Cross-Category object perception. As a pioneer of this area, Generalizable and Actionable Parts (GAParts) understanding has attracted increasing attention from relevant researchers. However, most recent works either have insufficient design regarding the symmetry issue or require rich symmetry annotation, which severely impedes precise GAPart pose estimation in data-lacking scenarios. In this paper, we propose SAFAG, a novel Symmetry Annotation-Free framework for Generalizable and Actionable Parts Pose Estimation. Specifically, we suggest a stepwise refinement two-stage framework for candidate-to-final quaternion regression, and tackle the symmetry prediction as a probability distribution problem with self-supervised learning strategy. The experimental results demonstrate the superior performance and robustness of our SAFAG. We believe that our work has the enormous potential to be applied in many areas of embodied AI system.
pose estimationgapartsself-supervised learningembodied ai
OpenReview ↗
PDF ↗
Code ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Generative Online Reinforcement Learning
Chubin Zhang, Zhenglin Wan, Feng Chen, Fuchao Yang, Lang Feng, Yaxin Zhou, Xingrui Yu, Yang You, Ivor Tsang, Bo An
Reinforcement learning (RL) faces a persistent tension: policies that are stable to optimize (e.g., Gaussians) are often too simple to represent the multimodal action distributions required for complex control. Conversely, expressive generative policies—such as diffusion and flow matching—can be difficult to optimize in online RL due to intractable likelihoods and gradients propagating through long sampling chains. We address this tension with a key structural principle: *decoupling optimization from generation*. Building on this, we introduce GoRL (Generative Online Reinforcement Learning), an algorithm-agnostic framework that trains expressive policies from scratch by confining policy optimization to a tractable latent space while delegating action synthesis to a conditional generative decoder. Using a two-timescale alternating schedule and anchoring decoder refinement to a fixed prior, GoRL enables stable optimization while continuously expanding expressiveness. Empirically, GoRL consistently outperforms unimodal and generative baselines across diverse continuous-control tasks. Notably, GoRL achieves returns exceeding **870** on HopperStand, more than $3\times$ the strongest baseline; on high-dimensional humanoid tasks, it further outperforms the strongest non-GoRL baseline by over an order of magnitude.
online-rldiffusion-policyflow-matchinggenerative-policy
OpenReview ↗
PDF ↗
Poster Session 8 (Hall A) · Thursday, July 9, 5:00-6:45 Geometry-Guided Modeling of Foundation Features Enables Generalizable Object Shape Deformation Learning
Yiyao Ma, Kai Chen, Zhongxiang Zhou, Zhuheng Song, Dongsheng Xie, Zelong Tan, Rong Xiong, Qi Dou
Monocular 3D shape recovery is fundamental to geometric understanding, yet achieving robust generalization across arbitrary viewpoints and unseen object categories remains a significant challenge. In this paper, we present a generalizable deformation learning framework that reconstructs 3D objects by explicitly deforming a category-level shape template to match the target observation. To address complex shape variations between the template and the target, we introduce a geometry-guided feature modeling mechanism. This process first enriches foundation features with template topology to yield a geometry-aware representation, which is then explicitly correlated with the target observation to guide precise deformation. Furthermore, to bridge the disparity between the fixed template and arbitrary target views, we propose a view-adaptive feature aggregation module. This module leverages multi-view template features and their corresponding camera poses to enrich the canonical template representation, ensuring robust feature alignment regardless of the target's perspective. Extensive experiments demonstrate that our approach significantly outperforms state-of-the-art methods in handling large shape variations and diverse viewpoints, exhibiting strong generalization to novel categories and effectively supporting downstream real-world dexterous robotic manipulation tasks. Project homepage: https://GODeform.github.io/
shape deformationfoundation modelsdexterous manipulation3d perception
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 GeoMoLa: Geometry-Aware Motion Latents for Learning Robust Manipulation Policies
Yunchao Zhang, Yijia Weng, Ruizhe Liu, Ming Hu, Leonidas Guibas, Yanchao Yang
Learning motion latents for robotic manipulation heavily relies on extracting motion patterns from visual sequences, yet effective action abstractions require understanding three-dimensional geometric transformations. Here, we introduce GeoMoLa (Geometry-Aware Motion Latents), which learns discrete motion latent codes by predicting how point clouds evolve during manipulation rather than reconstructing visual observations. This four-dimensional objective – spatial geometry changing through time – forces latent representations to encode actual physical motion rather than appearance patterns. GeoMoLa achieves state-of-the-art performance using only single-view RGB-D input, while existing methods require multi-view reconstruction, succeeding across diverse manipulation benchmarks. Our ablations reveal that geometric prediction is the key to driving performance, quantitatively validating that manipulation depends on spatial understanding. Furthermore, the learned codes exhibit effective motion abstraction: applying them to novel scenes produces physically consistent transformations regardless of visual context. Our real-world experiments also confirm this robustness capability, achieving robust manipulation with minimal demonstrations in cluttered environments where geometric reasoning determines success. Thus, we demonstrate that effective motion latents for robot control can better emerge from understanding motion through its three-dimensional effects rather than pixel-level patterns.
motion latentspoint cloudmanipulation policyreal robot
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Great Minds Think Alike: Contextual Tacit Communication for Decentralized LLM-Agent Cooperation
Yue Pei, Hongming Zhang, Jiarui Guan, Jusheng Zhang, Liang Lin, Haogang Zhu, Ziliang Chen
Large language models (LLMs) are increasingly used as planners for cooperative embodied agents, but multi-agent settings amplify inconsistency under partial observability and make explicit communication costly or even unavailable. Many existing approaches rely on online message passing; when communication is removed, agents often fall back to independent local planning that suffers from miscoordination. We introduce Contextual Tacit Communication, a gradient-free protocol that aligns decentralized decisions with a joint LLM value score without explicit message actions. Our method measures context-conditioned value rectifications via residual banding to pinpoint miscoordination actions and amortizes the resulting coordination signals into a retrieval-augmented Tacit Rule Memory that provides prompt-level cooperation rules at execution time. Experiments on VIKI, C-WAH, and TDW-MAT show that our approach improves cooperation performance over baselines while reducing runtime overhead compared with communication-based methods.
multi-agent cooperationllm planningembodied agentsdecentralized coordination
OpenReview ↗
PDF ↗
Poster Session 1 (Hall A) · Tuesday, July 7, 10:30-12:15 HALO: A Unified Vision-Language-Action Model for Embodied Multimodal Chain-of-Thought Reasoning
Quanxin Shou, Fangqi Zhu, Shuang Chen, Puxin Yan, Zhengyang Yan, Yikun Miao, Xiaoyi Pang, Zicong Hong, Ruikai Shi, Hao HUANG, Jie ZHANG, Song Guo
Vision–Language–Action (VLA) models perform well in robotic manipulation, but often struggle in long-horizon or out-of-distribution scenarios due to the lack of explicit mechanisms for multimodal reasoning and anticipating how the world evolves under action.
Recent works introduce textual chain-of-thought or visual subgoal prediction within VLA models, but still fail to offer a unified human-like framework for joint textual reasoning, visual foresight, and action prediction.
To this end, we propose HALO, a unified VLA model that enables embodied multimodal chain-of-thought (EM-CoT) reasoning through textual reasoning, fine-grained visual subgoal prediction, and EM-CoT-augmented action prediction.
We instantiate HALO with a Mixture-of-Transformers (MoT) architecture that decouples semantic reasoning, visual foresight, and action prediction into specialized experts with seamless cross-expert collaboration.
To enable HALO learning at scale, we introduce an automated pipeline to synthesize EM-CoT training data along with a carefully crafted training recipe.
Extensive experiments demonstrate that: (1) HALO achieves superior performance in both simulated and real world, surpassing baseline policy $\pi_0$ by 25.9\% on the RoboTwin benchmark; (2) all proposed components of the training recipe and EM-CoT design help improve task success rate; and (3) HALO exhibits strong generalization under aggressive unseen environment randomization with our proposed EM-CoT reasoning.
chain-of-thoughtmixture-of-transformersrobotwinreal robot
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Harmonized Dual Policy Improvement for Modelic Reinforcement Learning
Guojian Zhan, Likun Wang, Feihong Zhang, Yang Guan, Shengbo Eben Li
Policy-planner bootstrapping has emerged as a powerful paradigm in model-based reinforcement learning (MBRL). We formalize this process as a dual policy improvement mechanism synergizing: (i) exploitative improvement via off-policy $Q$-maximization, and (ii) lookahead improvement via planner alignment. While we theoretically prove that these improvements anchor to the same optimum, practical training process inevitably encounters gradient disagreement. Exacerbated by approximation inaccuracies and non-stationary data, this disagreement induces destructive interference in policy updates, destabilizing the bootstrapping loop and leading to suboptimal convergence. To address this, we propose harmonized dual policy improvement (HDPI), a gradient-level framework that reconciles exploitative and lookahead improvements through a harmonic optimization scheme. This scheme effectively maximizes the worst-case inner product between the harmonized update and the original gradients, ensuring directional consistency and stabilizing policy evolution. Extensive empirical evaluations on 14 challenging tasks from the DeepMind Control Suite and the Humanoid-Bench demonstrate that HDPI significantly enhances training stability and asymptotic performance, outperforming a wide range of strong baselines.
model-based-rlpolicy-optimizationhumanoidbenchplanning
OpenReview ↗
PDF ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 HDFlow: Hierarchical Diffusion-Flow Planning for Long-horizon Tasks
Gireesh Nandiraju, Yuanliang Ju, Chaoyi Xu, Weiheng Liu, Yuxuan Wan, He Wang
Recent advances in generative models have shown promise in generating behavior plans for long-horizon, sparse reward tasks. While these approaches have achieved promising results, they often lack a principled framework for hierarchical decomposition and struggle with the computational demands of real-time execution, due to their iterative denoising process. In this work, we introduce $\textbf{Hierarchical Diffusion-Flow}$ ($\texttt{\textbf{HDFlow}}$), a novel hierarchical planning framework that optimally leverages the strengths of $\textit{diffusion}$ and $\textit{rectified flow}$ models to overcome the limitations of single-paradigm generative planners. $\texttt{\textbf{HDFlow}}$ employs a high-level diffusion planner to generate sequences of strategic subgoals in a learned latent space, capitalizing on diffusion's powerful exploratory capabilities. These subgoals then guide a low-level rectified flow planner that generates smooth and dense trajectories, exploiting the speed and efficiency of ordinary differential equation (ODE)-based trajectory generation. We evaluate $\texttt{\textbf{HDFlow}}$ on four challenging furniture assembly tasks in both simulation and real-world, where it significantly outperforms state-of-the-art methods. Furthermore, we also showcase our method's generalizability on two long-horizon benchmarks comprising diverse locomotion and manipulation tasks. Project website: https://hdflow-page.github.io/
hierarchical planningdiffusionflow matchingfurniture assemblylong-horizon
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Hierarchical Goal Abstractions via Learned Subset Relations
Fabian Wurzberger, Sebastian Gottwald, Zeqiang Zhang, Daniel Alexander Braun
In self-supervised goal-conditioned reinforcement learning (RL) without external rewards, goals are typically specified by observations sampled from experience.
However, depending on the observation structure, such a fixed representation of goals may be either too concrete (requiring exact pixel-level matches) or too abstract (involving ambiguous observations).
Here we propose the construction of hierarchical latent goal spaces that integrate both concrete and abstract goals.
To this end, we use an energy function to learn a partially ordered space, in which a subset relation between observations naturally induces a hierarchy from concrete to abstract goals.
This representation enables agents to disambiguate specific states while also generalizing to shared concepts.
In experiments on navigation and robotic manipulation, agents trained with our hierarchical goal space achieve higher task success and greater generalization to novel tasks compared to agents limited to purely observational goals.
goal-conditioned rlhierarchical goalsself-supervised rlnavigation
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Hierarchical Policy Learning via Spectral Decomposition
Shuxin Cao, Liquan Wang, Walker Byrnes, Yiye Chen, Yilun Du, Animesh Garg
In this paper, we identify a semantic decomposition in robot action sequences, separating task-level motion intent from execution-level refinements.
By analyzing actions in the spectral domain using the discrete cosine transform (DCT), we observe that low-frequency components capture global motion trajectories, while high-frequency components encode precise timing, alignment, and contact behaviors.
Motivated by this structure, we propose Causal Spectral Policy (CSP), which models action generation as a causal coarse-to-fine process: coarse motion is predicted from observation and language, and fine corrections are generated conditionally on the realized trajectory.
Across simulation and real-world evaluations, CSP consistently outperforms strong baselines on precision-sensitive manipulation tasks.
Additionally, we propose human-inspired teleoperation noise injection as a data augmentation method under which our approach demonstrates strong robustness to noisy demonstrations
imitation learningspectral decompositionteleoperationprecision manipulation
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 HiMe: Hierarchical Embodied Memory for Long-Horizon Vision-Language-Action Control
Li Ji, Siyin Wang, Pengfang Qian, Xiaopeng Yu, Yihai Tian, Zhaoye Fei, Jingjing Gong, Xipeng Qiu
Current Vision-Language-Action (VLA) models excel at robotic manipulation but often struggle with non-Markovian tasks requiring long-term memory and reasoning due to their reliance on immediate observations. Existing solutions face a frequency-competence paradox, where high-performance models are too slow for real-time control, while faster models lack sufficient reasoning capabilities. To resolve this architectural misalignment, we propose **HiMe**, a Hierarchical Embodied Memory framework that decouples embodied intelligence into a high-frequency Executor for execution, a Sentry for working memory, and a Planner for long-term strategy. We also introduce a dynamic knowledge system based on cross-modal semantic schemas and active management mechanisms, allowing robots to maintain memory plasticity through "Add, Update, and Delete" operations. This hierarchical design effectively balances the conflict between real-time execution and slow thinking planning, significantly improving success rates in long-horizon tasks. Experiments demonstrate that this approach not only outperforms flat memory baselines but also exhibits the novel ability to self-correct its internal knowledge based on human preferences.
hierarchical memorylong-horizon controlvla
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Human-in-the-Loop Policy Optimization for Preference-Based Multi-Objective Reinforcement Learning
Tianmeng Hu, Biao Luo, Ke Li
Multi-objective reinforcement learning (MORL) seeks policies that effectively balance conflicting objectives. However, presenting many diverse policies without accounting for the decision maker’s (DM’s) preferences can overwhelm the decision-making process. On the other hand, accurately specifying preferences in advance is often unrealistic. To address these challenges, we introduce a human-in-the-loop MORL framework that interactively discovers preferred policies during optimization. Our approach proactively learns the DM’s implicit preferences in real time, requiring no a priori knowledge. Importantly, we integrate this preference learning directly into a parallel optimization framework, balancing exploration and exploitation to identify high-quality policies aligned with the DM's preferences. Evaluations on a complex quadrupedal robot simulation environment demonstrate that, with only interactions, our proposed method can identify policies aligned with human preferences, e.g., running like a dog. Further experiments on seven MuJoCo tasks and a multi-microgrid system design task against eight state-of-the-art MORL algorithms fully demonstrate the effectiveness of our proposed framework. Demonstrations and full experiments are in https://sites.google.com/view/pbmorl/home.
multi-objective rlquadrupedpreference learninghuman-in-the-loop
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 Hydra-Nav: Object Navigation via Adaptive Dual-Process Reasoning
Zixuan Wang, Huang Fang, Shaoan Wang, Yuanfei Luo, Heng Dong, Wei Li, Yiming Gan
While large vision-language models (VLMs) show promise for object goal navigation, current methods still struggle with low success rates and inefficient localization of unseen objects—failures primarily attributed to weak temporal-spatial reasoning. Meanwhile, recent attempts to inject reasoning into VLM-based agents improve success rates but incur substantial computational overhead. To address both the ineffectiveness and inefficiency of existing approaches, we introduce Hydra-Nav, a unified VLM architecture that adaptively switches between a deliberative "slow system" for analyzing exploration history and formulating high-level plans, and a reactive "fast system" for efficient execution. We train Hydra-Nav through a three-stage curriculum: (i) spatial-action alignment to strengthen trajectory planning, (ii) memory-reasoning integration to enhance temporal-spatial reasoning over long-horizon exploration, and (iii) iterative rejection fine-tuning to enable selective reasoning at critical decision points. Extensive experiments demonstrate that Hydra-Nav achieves state-of-the-art performance on the HM3D, MP3D, and OVON benchmarks, outperforming the second-best methods by 11.1\%, 17.4\%, and 21.2\%, respectively. Furthermore, we introduce SOT (Success weighted by Operation Time), a new metric to measure search efficiency across VLMs with varying reasoning intensity. Results show that adaptive reasoning significantly enhances search efficiency over fixed-frequency baselines. Project page is available at: https://zixuan-wang99.github.io/Hydra-Nav/.
object goal navigationvlmdual-process reasoningembodied navigation
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Improving Diffusion Planners by Self-Supervised Action Gating with Energies
Yuan Lu, Dongqi Han, Yansen Wang, Dongsheng Li
Diffusion planners are a strong approach for offline reinforcement learning, but they can fail when value-guided selection favours trajectories that score well yet are locally inconsistent with the environment dynamics, resulting in brittle execution. We propose Self-supervised Action Gating with Energies (SAGE), an inference-time re-ranking method that penalises dynamically inconsistent plans using a latent consistency signal. SAGE trains a Joint-Embedding Predictive Architecture (JEPA) encoder on offline state sequences and an action-conditioned latent predictor for short horizon transitions. At test time, SAGE assigns each sampled candidate an energy given by its latent prediction error and combines this feasibility score with value estimates to select actions. SAGE can integrates into existing diffusion planning pipelines that can sample trajectories and select actions via value scoring; it requires no environment rollouts and no policy re-training. Across locomotion, navigation, and manipulation benchmarks, SAGE improves the performance and robustness of diffusion planners.
diffusion planningoffline rlenergy-basedmanipulation
OpenReview ↗
PDF ↗
Poster Session 7 (Hall A) · Thursday, July 9, 2:30-4:15 InfoGeo: Information-Theoretic Object-Centric Learning for Cross-View Generalizable UAV Geo-Localization
Hongyang ZHANG, Maonan Wang, Ziyao Wang, Hongrui Yin, Man On Pun
Cross-view geo-localization (CVGL) is fundamental for precise localization and navigation in GPS-denied environments, aiming to match ground or UAV imagery with satellite views. Existing approaches often rely on global feature alignment, but they suffer from substantial domain shifts induced by varying regional textures and weather conditions. This issue becomes even more pronounced in UAV-based scenarios, where the broader perspective inevitably introduces dense, fine-grained objects, creating significant visual clutter. To address this, we draw inspiration from Object-Centric Learning (OCL) and propose InfoGeo, an information-theoretic framework designed to enhance robustness and generalization. InfoGeo reformulates the optimization as an information bottleneck process with two core objectives: (i) maximizing view-invariant information by aligning the object-centric structural relations across views, and (ii) minimizing view-specific noisy signals through cross-view knowledge constraints. Extensive evaluations across diverse benchmarks and challenging scenarios demonstrate that InfoGeo significantly outperforms state-of-the-art methods.
uavgeo-localizationcross-viewobject-centric learning
OpenReview ↗
PDF ↗
Code ↗
Poster Session 1 (Hall A) · Tuesday, July 7, 10:30-12:15 Instruction Decomposition and Action Alignment for Vision-Language Navigation
Zihao Xin, Wentong Li, Yixuan Jiang, Bin Wang, Piji Li, Jianke Zhu, Jie Qin, Sheng-Jun Huang
Vision-and-Language Navigation (VLN) empowered by Multimodal Large Language Models (MLLMs) is promise, yet remains challenged by long-horizon tasks with complex user instructions. Existing approaches that continuously condition on full instructions incur high latency due to abundant visual tokens and exacerbates instruction interference, where irrelevant text noise induces hallucinations.
To address these limitations, we propose IDEAL-VLN ( \textbf{I}nstruction \textbf{DE}composition and \textbf{A}ction a\textbf{L}ignment ), a novel paradigm that reformulates navigation as a causal inference chain. We decompose the task into two sequential steps: Semantic Anchoring and Action Alignment. We adopt a \textit{Think-Before-Act} mechanism that first infers the immediate semantic anchor from the global context and then generates actions conditioned solely on this anchor. This design constructs an explicit information bottleneck, suppressing spurious correlations from irrelevant instruction. Moreover, to alleviate cognitive collapse and limited exploration during training, we introduce a hierarchical correction framework that combines semantic-level thought correction with a spatially-aware adaptive intervention strategy. This strategy adjusts expert intervention probability based on geodesic distance, effectively defining a semantic safety boundary. To support this paradigm, we contribute the Instruction-Aligned Navigation Dataset containing 160K image-text pairs.
Extensive experiments demonstrate that IDEAL-VLN achieves state-of-the-art performance and robustness across major benchmarks while significantly reducing inference costs.
vision-and-language navigationinstruction followingmllm
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Joint Navigation and Manipulation Planning with 3D Interaction Chains
Keming Zhang, Sixian Zhang, Xinhang Song, Hongyu Wang, Yiyao Wang, Yingjie Wang, Shuqiang Jiang
Open-vocabulary mobile manipulation (OVMM) requires long-horizon navigation in unseen environments and object-centric manipulation. Most existing methods treat navigation and manipulation as separate stages, which can yield navigation endpoints that are poor for manipulation or manipulation-friendly poses that are globally inefficient.
We address this mismatch with 3D Interaction Chains (3D-IC), a unified framework that couples multi-stage navigation and manipulation planning. 3D-IC maintains a shared 3D feature map for both skills, generates stage-aligned interaction waypoints, and links them into candidate multi-stage chains. A hierarchical policy then scores these chains by jointly considering feasibility (via VLM reasoning over waypoint-centric 3D features) and transition cost, selecting the best trade-off between success and path efficiency. The robot executes the next waypoint and replans as new observations arrive. Experiments in simulation and on a real Stretch 3 robot demonstrate consistent gains in both task success and trajectory efficiency.
mobile manipulationnavigation planningreal robot3d perception
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Joint-Space Empowerment as a Theory of Dexterous Motor Coordination
James Heald, Vittorio Caggiano, Vikash Kumar, Maneesh Sahani
Searching for effective policies is notoriously challenging in high-dimensional musculoskeletal systems, where multiple muscles actuate individual joints. Although this redundancy complicates naive policy search, it also implies that effective control can be captured by a low-dimensional action manifold. To identify such a manifold, we introduce Joint-Space Empowerment (JoSE), a novel information-theoretic objective that quantifies how much control an agent has over its mechanical degrees-of-freedom. We frame manifold discovery as an optimal precoding problem—where a state-dependent precoder maps low-dimensional latent actions to high-dimensional muscle commands—and derive the optimal precoder in closed form under control-affine Gaussian dynamics. We show that manipulation policies learned on this manifold display significantly enhanced dexterity, sample efficiency, and improved generalization. These results present optimal precoding as a general information-theoretic paradigm for coordinating high-dimensional actuators to control low-dimensional features.
dexterous manipulationmuscle synergiesinformation theorymotor control
OpenReview ↗
PDF ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 Knothe-Rosenblatt Quantile Regression for Risk-sensitive Multi-objective Reinforcement Learning
Gwangpyo Yoo, Woo Kyung Kim, Honguk Woo
In this work, we extend distributional reinforcement learning (RL) to develop a risk-sensitive multi-objective RL framework, with applications to domains such as finance and robotics. We achieve this by adopting vector-risk measures and approximating them via Knothe-Rosenblatt (KR) quantile regression. This approach directly extends the IQN framework to the multi-objective setting, aligns with the axiomatic definition of vector-risk measures, and guarantees that critics converge under the distributional Bellman operator. To mitigate the artificial ordering imposed by the KR map, we employ a transformer architecture without positional encoding, and introduce MO-TQC for training stability. We demonstrate improved performance on MO-Gymnasium benchmarks and use our framework to study risk-sensitive policies in multi-objective tasks.
multi-objective-rlrisk-sensitivedistributional-rlquantile-regression
OpenReview ↗
PDF ↗
Code ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 Knowledge Diversion for Efficient Morphology Control and Policy Transfer
Fu Feng, Ruixiao Shi, Yucheng Xie, Jianlu Shen, Jing Wang, Xin Geng
Universal morphology control aims to learn a universal policy that generalizes across heterogeneous robot morphologies, with Transformer-based controllers emerging as a dominant choice. However, such architectures incur substantial computational costs, resulting in high deployment overhead, and existing methods exhibit limited cross-task generalization, necessitating training from scratch for each new task. To this end, we propose DivMorph, a modular training paradigm that leverages knowledge diversion to learn *decomposable controllers*. DivMorph factorizes randomly initialized Transformer weights into *basic knowledge units* via SVD and employs dynamic soft gating, conditioned on task and morphology embeddings, to adaptively modulate these units into universal *learngenes* and morphology- and task-specific *tailors* during training, thereby achieving knowledge disentanglement. By selectively activating relevant components, DivMorph adaptively recomposes the controller, enabling efficient policy deployment and effective policy transfer to novel tasks. Extensive experiments demonstrate that DivMorph achieves state-of-the-art performance, improving sample efficiency for cross-task transfer by 3.3$\times$ and reducing model size for single-agent deployment by 16.7$\times$.
morphology controlmodular robotspolicy transfertransformers
OpenReview ↗
PDF ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 LabBuilder: Protocol-Grounded 3D Layout Generation for Interactable and Safe Laboratory
Jianbao Cao, zhangrui zhao, Bohan Feng, Zixuan Hu, Rui Li, Haiyuan Wan, Chenxi Li, jingyuan Li, Wenzhe Cai, LEI BAI, Wanli Ouyang, LINGYU DUAN, Di Huang, Minting Pan, Sha Zhang, Xinzhu Ma, SHIXIANG TANG, Dongzhan Zhou
Automated laboratories hold the promise of accelerating scientific discovery, yet their deployment is bottlenecked by the difficulty of designing safe and executable environments. While simulator-based design offers scalability, existing 3D scene generation methods are primarily tailored for household settings, optimizing for visual plausibility while neglecting the protocol grounding and layout-level safety constraints essential for scientific experimentation. We present LabBuilder, an end-to-end system that generates and verifies 3D laboratory layouts from concise textual specifications. It operates through three tightly coupled components: LabForge first curates a meta-dataset of annotated assets and chemical knowledge, translating natural language specifications into structured protocols; building on these protocols, LabGen synthesizes laboratory layouts via an iterative, constraint-aware optimization strategy; finally, LabTouchstone evaluates the resulting layouts as a unified benchmark. Extensive experiments demonstrate that LabBuilder significantly outperforms existing state-of-the-art methods, producing laboratory environments that are realistic and valid under modeled geometric, chemical-safety, and navigation constraints.
lab-automationscene-generationsafety-verificationlayout-generation
OpenReview ↗
PDF ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 LAGEA: Language Guided Embodied Agents for Robotic Manipulation
Abdul Monaf Chowdhury, Akm Moshiur Rahman Mazumder, Safaeid Hossain Arib, Rabeya Akter
Robotic manipulation benefits from foundation models that describe goals, but today's agents still lack a principled way to learn from their own mistakes. We ask whether natural language can serve as feedback, an error-reasoning signal that helps embodied agents diagnose what went wrong and correct course. We introduce LAGEA (Language Guided Embodied Agents), a framework that turns episodic, schema-constrained reflections from a vision language model (VLM) into temporally grounded guidance for reinforcement learning. LAGEA summarizes each attempt in concise language, localizes the decisive moments in the trajectory, aligns feedback with visual state in a shared representation, and converts goal progress and feedback agreement into bounded, step-wise shaping rewardswhose influence is modulated by an adaptive, failure-aware coefficient. This design yields dense signals early when exploration needs direction and gracefully recedes as competence grows. On the Meta-World MT10 and Robotic Fetch embodied manipulation benchmarks, LAGEA improves average success over the state-of-the-art (SOTA) methods by 9.0% on random goals, 5.3% on fixed goals, and 17% on fetch tasks, while converging faster. These results support our hypothesis: language, when structured and grounded in time, is an effective mechanism for teaching robots to self-reflect on mistakes and make better choices. Code will be released soon.
language-guided rlvlm feedbackreward shapingrobotic manipulation
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Lagrangian Perturbation Diffusion Steering: Latent Reinforcement Learning for Generative Policies
Hikmet Simsir, Ozgur S. Oguz
Behavior cloning with high-capacity generative policies achieves strong imitation performance, but is often limited by demonstration coverage and distribution shift. Direct reinforcement learning fine-tuning can improve performance, but updating large action decoders is frequently unstable and sample inefficient. We propose **Lagrangian Perturbation Diffusion Steering (LP-DS)**, a lightweight adaptation method that improves a frozen generative policy by learning a compact noise-space perturbation before decoding. LP-DS optimizes this perturbation with a Lagrangian trust-region objective, improving downstream value while constraining deviation from the latent prior. Across RoboMimic manipulation, OpenAI Gym locomotion, and Adroit dexterous manipulation benchmarks, LP-DS improves sample efficiency, success, and return while maintaining higher action-space entropy than unconstrained noise-space steering, with return improvements of up to 25\% over prior baselines. Additional evaluations with flow-matching backbones, a large vision-language-action model, and physical Franka deployment show that LP-DS is not limited to compact diffusion policies or simulated benchmarks. Project page: https://sites.google.com/view/lp-ds/home.
diffusion policyrl fine-tuningdexterous manipulationreal robot
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 Langevin Rollout Optimization for Modelic Reinforcement Learning
Tianyi Zhang, Likun Wang, Guojian Zhan, Feihong Zhang, Yang Guan, Yao Lyu, Shengbo Eben Li
Planning-driven model-based (modelic) reinforcement learning has achieved impressive success in continuous control tasks but predominantly relies on zero-order optimizers like Model Predictive Path Integral (MPPI).
While robust for global exploration, MPPI updates actions solely through sampling and neglects the smooth return gradients inherent in structured dynamics that guide fine-grained search.
To complement MPPI’s robustness with gradient-guided precision, we first propose \textbf{La}ngevin \textbf{R}ollout \textbf{O}ptimization (LaRO), which leverages return gradients to refine actions via Langevin dynamics, achieving reliable local convergence without sacrificing multimodal exploration.
This is supported by a score-augmented world model that jointly learns dynamics and a score function within a unified latent space, facilitating efficient and accurate gradient estimation for real-time planning.
Second, we combine MPPI and LaRO through a simple yet effective choice mechanism, termed \textbf{M}aximum \textbf{L}ook-\textbf{A}head \textbf{P}lanning (MLAP).
Finally, we instantiate MLAP within the latest BOOM algorithm, replacing its MPPI-only planner and yielding BOOM-L. Empirical results on the DeepMind Control Suite and Humanoid Bench demonstrate that BOOM-L consistently outperforms strong baselines in both sample efficiency and final performance.
model-based-rlmppilangevin-dynamicsplanning
OpenReview ↗
PDF ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 LangForce: Bayesian Decomposition of Vision Language Action Models via Latent Action Queries
Shijie Lian, Bin Yu, Xiaopeng LIN, Laurence Tianruo Yang, Zhaolong Shen, Changti Wu, YuZhuo Miao, Cong Huang, Kai Chen
Vision-Language-Action (VLA) models have shown promise in robot manipulation but often struggle to generalize to new instructions or complex multi-task scenarios. We identify a critical pathology in current training paradigms where goal-driven data collection creates a dataset bias. In such datasets, language instructions are highly predictable from visual observations alone, causing the conditional mutual information between instructions and actions to vanish, a phenomenon we term Information Collapse. Consequently, models degenerate into vision-only policies that ignore language constraints. To address this, we propose LangForce, enforces instruction following via Bayesian decomposition. By introducing learnable Latent Action Queries, we construct a dual-branch architecture to estimate both a vision-only prior $p(a \mid v)$ and a language-conditioned posterior $\pi(a \mid v, \ell)$. We then optimize the policy to maximize the conditional Pointwise Mutual Information (PMI) between actions and instructions. This objective effectively penalizes the vision shortcut and rewards actions that explicitly explain the language command. Extensive experiments across on three benchmarks demonstrate substantial gains, including an 11.3\% improvement on the challenging OOD SimplerEnv benchmark, validating the ability of LangForce to robustly ground language in action.
vlabayesian decompositioninstruction grounding
OpenReview ↗
PDF ↗
Code ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 LARA: Latent Action Representation Alignment for Vision-Language-Action Models
Mengya Liu, Baoxiong Jia, Jiangyong Huang, Jingze Zhang, Siyuan Huang
Visual-language-action (VLA) models enable robots to predict actions directly from observations and language instructions, but their performance depends on large-scale, high-quality data and is limited by the scarcity of real-world robot action datasets. To facilitate VLA model learning with abundant unlabeled human videos, Latent Action Models (LAM) learn latent action representations from visual dynamics to provide additional supervision for VLA learning. However, LAM and VLA are typically trained separately, leaving LAM ungrounded during VLA training and VLA models constrained by frozen LAM representations. To address these issues, we propose Latent Action Representation Alignment (LARA), a plug-and-play framework that jointly optimizes LAM and VLA via representation alignment. This enables reciprocal benefits where LAMs learn with action trajectories to avoid spurious visual changes, while VLAs are regularized by forward dynamics learned within LAMs to reduce hallucinations of functionally ineffective trajectories. We demonstrate LARA's versatility and effectiveness for pre-training, post-training enhancement of pre-trained VLA models, and LAM refinement, achieving an average of ~10%, ~5%, and ~15% improvement over 3 simulation and 1 meticulously designed real-world robotic manipulation benchmarks. The code is publicly available at https://github.com/lmy1001/LARA.
latent action modelvlarepresentation alignmentrobot manipulation
OpenReview ↗
PDF ↗
Code ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 LASER: Learning Active Sensing for Continuum Field Reconstruction
Huayu Deng, Jinghui Zhong, Xiangming Zhu, Yunbo Wang, Xiaokang Yang
High-fidelity measurements of continuum physical fields are essential for scientific discovery and engineering design but remain challenging under sparse and constrained sensing. Conventional reconstruction methods typically rely on fixed sensor layouts, which cannot adapt to evolving physical states. We propose LASER, a unified, closed-loop framework that formulates active sensing as a Partially Observable Markov Decision Process (POMDP). At its core, LASER employs a continuum field latent world model that captures the underlying physical dynamics and provides intrinsic reward feedback. This enables a reinforcement learning policy to simulate ''what-if'' sensing scenarios within a latent imagination space. By conditioning sensor movements on predicted latent states, LASER navigates toward potentially high-information regions beyond current observations. Our experiments demonstrate that LASER consistently outperforms static and offline-optimized strategies, achieving high-fidelity reconstruction under sparsity across diverse continuum fields.
active-sensingpomdpreinforcement-learningfield-reconstruction
OpenReview ↗
PDF ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 LaST$_{0}$: Latent Spatio-Temporal Chain-of-Thought for Robotic Vision-Language-Action Model
Zhuoyang Liu, Jiaming Liu, Hao Chen, Jiale Yu, Ziyu Guo, Chengkai Hou, Chenyang Gu, Xiangju Mi, Renrui Zhang, Kun Wu, Zhengping Che, Jian Tang, Pheng-Ann Heng, Shanghang Zhang
Vision-Language-Action (VLA) models have recently shown strong generalization, with some approaches seeking to explicitly generate linguistic reasoning traces or predict future observations prior to execution. However, explicit reasoning typically incurs non-negligible inference latency, which constrains the temporal resolution required for robotic manipulation. Moreover, such reasoning is confined to the linguistic space, imposing a representational bottleneck that struggles to faithfully capture ineffable physical attributes. To mitigate these limitations, we propose LaST$_0$, a framework that enables efficient reasoning before acting through a Latent Spatio-Temporal Chain-of-Thought (CoT), capturing fine-grained physical and robotic dynamics that are often difficult to verbalize. Specifically, we introduce a token-efficient latent CoT space that models future visual dynamics, 3D structural information, and robot proprioceptive states, and further extends these representations across time to enable temporally consistent implicit reasoning trajectories. Furthermore, LaST$_0$ adopts a dual-system architecture implemented via a Mixture-of-Transformers design, where a reasoning expert conducts low-frequency latent inference and an acting expert generates high-frequency actions conditioned on robotics-oriented latent representations. To facilitate coordination, LaST$_0$ is trained with heterogeneous operation frequencies, enabling adaptive switching during deployment. Across 10 real-world tasks spanning tabletop, mobile, and dexterous hand manipulation, LaST$_0$ improves mean success rates by 13%, 14% and 14% over prior SOTA VLA methods, respectively.
latent chain-of-thoughtvladual-systemreal robot
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Last-iterate Convergence of ADMM on Multi-affine Quadratic Equality Constrained Problem
Yutong Chao, Michal Ciebielski, Jalal Etesami, Majid Khadiv
In this paper, we study a class of non-convex optimization problems known as multi-affine quadratic equality constrained problems, which appear in various applications--from generating feasible force trajectories in robotic locomotion and manipulation to training neural networks. Although these problems are generally non-convex, they exhibit convexity or related properties when all variables except one are fixed. Under mild assumptions, we prove that the alternating direction method of multipliers (ADMM) converges when applied to this class of problems. Furthermore, when the "degree" of non-convexity in the constraints remains within certain bounds, we show that ADMM achieves a linear convergence rate. We validate our theoretical results through practical examples in robotic locomotion.
optimizationrobot locomotionforce trajectoriesadmm
OpenReview ↗
PDF ↗
Poster Session 7 (Hall A) · Thursday, July 9, 2:30-4:15 LAST: Bridging Vision-Language and Action Manifolds via Gromov-Wasserstein Alignment
Huaihai Lyu, Chaofan Chen, Yuheng Ji, Xiansheng Chen, Pengwei Wang, Shanghang Zhang, Changsheng Xu
We formulate the learning of generalist Vision-Language-Action (VLA) models as a Gromov-Wasserstein alignment problem, aiming to map semantically similar VL embeddings to physically similar motion primitives. However, solving this is challenging due to the mathematical heterogeneity between the domains: the semantic space of vision-language is topologically linear and isotropic, while the physical manifold of robotic action is non-Euclidean and anisotropic. As a result, direct regression approaches fail due to the disjoint metric structures of these domains, making standard distance minimization ill-posed.
To resolve this incompatibility, we introduce LAST (Lie-algebraic Action Space Tokenizer). LAST reconstructs the action space to establish a more consistent metric alignment between the VL and Action modalities. Specifically, LAST bridges the heterogeneity via two stages: (1) *Global Topological Linearization*, which linearizes the action manifold through Lie-algebraic mapping, converting trajectories into a fixed-length, physically additive representation; and (2) *Local Metric Discretization*, where the representation is discretized hierarchically into schemas and whitened residuals, establishing a mathematical isomorphism with the isotropic Euclidean metric. By addressing the structural mismatch globally and locally, LAST enables VLA models with enhanced convergence and generalizability.
vlagromov-wassersteinaction tokenizerrepresentation learning
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Latent Reasoning VLA: Latent Thinking and Prediction for Vision-Language-Action Models
Shuanghao Bai, Jing Lyu, Wanqi Zhou, Zhe Li, Dakai Wang, Lei Xing, Xiaoguang Zhao, Pengwei Wang, Zhongyuan Wang, Cheng Chi, Badong Chen, Shanghang Zhang
Vision-Language-Action (VLA) models benefit from Chain-of-Thought (CoT) reasoning, but existing approaches incur high inference overhead and rely on discrete reasoning representations that mismatch continuous perception and control.
We propose Latent Reasoning VLA (LaRA-VLA), a unified VLA framework that internalizes multi-modal CoT reasoning into continuous latent representations for embodied action.
LaRA-VLA performs unified reasoning and prediction in latent space, eliminating explicit CoT generation at inference time and enabling efficient, action-oriented control.
To realize latent embodied reasoning, we introduce a curriculum-based training paradigm that progressively transitions from explicit textual and visual CoT supervision to latent reasoning, and finally adapts latent reasoning dynamics to condition action generation.
We construct two structured CoT datasets, LIBERO-LaRA and Bridge-LaRA, and evaluate LaRA-VLA across simulation benchmarks and long-horizon real-robot manipulation tasks. Experimental results show that LaRA-VLA outperforms existing state-of-the-art VLA methods while achieving up to a 90\% reduction in inference latency compared to explicit CoT-based VLA approaches, highlighting latent reasoning as an effective and efficient paradigm for real-time embodied control.
latent reasoningvlareal robotchain-of-thought
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Learning 3D-Gaussian Simulators from RGB Videos
Mikel Zhobro, Andreas René Geist, Georg Martius
Realistic simulation is critical for applications ranging from robotics to animation. Video generation models have emerged as a way to capture real-world physics from data, but they often face challenges in maintaining spatial consistency and object permanence, relying on memory mechanisms to compensate. As a complementary direction, we present 3DGSim, a learned 3D simulator that directly learns physical interactions from multi-view RGB videos. 3DGSim adopts MVSplat to learn a latent particle-based representation of 3D scenes, a Point Transformer for the particle dynamics, a Temporal Merging module for consistent temporal aggregation, and Gaussian Splatting to produce novel view renderings. By jointly training inverse rendering and dynamics forecasting, 3DGSim embeds physical properties into point-wise latent features. This enables the model to capture diverse behaviors, from rigid and elastic to cloth-like dynamics and boundary conditions (e.g., fixed cloth corners), while producing realistic lighting effects. We show that 3DGSim can generate physically plausible results even in out of distribution cases, e.g. ground removal or multi-object interactions, despite being trained only on single-body collisions.
physics-simulationgaussian-splattingparticle-dynamicsworld-models
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 7 (Hall A) · Thursday, July 9, 2:30-4:15 Learning Anisotropic Value Geometry with Finsler Reinforcement Learning
Jumman Hossain, Nirmalya Roy
We introduce **Finslerian Reinforcement Learning (FiRL)**, an RL framework that makes directional costs explicit and improves robustness to tail risk. FiRL incorporates a *Finsler metric* into the locomotion cost, expressing effort as $F(x,v)$ that depends on the state $x$ and motion $v$, so it can capture uphill versus downhill asymmetry, lateral slip, and other direction-dependent effects. To handle rare but catastrophic outcomes, FiRL optimizes a Conditional Value-at-Risk ($CVaR_\alpha$) objective. We derive the corresponding risk-sensitive Bellman equation and show that the resulting CVaR--Finsler Bellman operator is a $\gamma$-contraction. This guarantees a unique fixed-point value function, while the underlying Finsler cost induces an asymmetric path cost $d_F$ that satisfies a triangle inequality despite directional asymmetry. We then develop a FiRL actor--critic algorithm to learn policies under this anisotropic, risk-averse objective. Across simulation benchmarks and real-world robot trials, FiRL demonstrates safer and more energy-efficient locomotion behavior than strong baselines such as risk-neutral PPO. For instance, on a $12^\circ$ sloped Hopper task, FiRL reduces worst-case ($CVaR_{0.1}$) impact forces by over 35% and total energy cost by 15%, while also improving success rate.
locomotionrisk-sensitive rlreal robot trialsfinsler geometry
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 Learning Attribute–Affordance Hierarchies in Hyperbolic Space for Open-Vocabulary 3D Object Affordance Grounding
Yuxuan Wang, Tong Li, Yihang Zhu, Guangtao Lyu, Yukuan Min, Chenghao Xu, Jiexi Yan, Xu Yang, Cheng Deng
This paper pays attention to open-vocabulary 3D object affordance grounding (OVAG), which aims to localize affordance regions on 3D objects by leveraging interaction images or textual instructions. Most existing methods treat interaction images as sources of external affordance knowledge and align them with 3D visual representations, while overlooking the intrinsic relationship between local object attributes and affordances, which limits localization accuracy and generalization. For instance, a cup handle affords grasping due to its curved shape and appropriate thickness, indicating that affordances emerge from specific attribute compositions rather than global object appearance.
Motivated by this, we propose Attribute-Affordance Hierarchies (AAH) learning framework that explicitly models the hierarchical relationships between object-region attributes and affordances. Our approach first captures local region relationships using hypergraph, and then projects these region-level concepts into a hyperbolic space to encode their hierarchical organization. Furthermore, we introduce counterfactual attribute samples to encourage robust learning of attribute–affordance dependencies under varying conditions. By jointly modeling visual structure and hierarchical concept information, our method achieves more accurate affordance localization. Extensive experiments and qualitative analyses demonstrate the effectiveness of our approach.
affordance grounding3d perceptionopen-vocabularygrasping
OpenReview ↗
PDF ↗
Poster Session 6 (Hall A) · Thursday, July 9, 10:30-12:15 Learning High-Frequency Continuous Action Chunks in Latent Space
Kunyun Wang, Yuhang Zheng, Yupeng Zheng, Jieru Zhao, Wenchao Ding
Modern robotic policies increasingly rely on action chunking to execute complex tasks in the physical world. While action chunking improves temporal consistency at moderate action frequencies, it becomes insufficient when the action frequency is further increased (e.g., to 60~Hz). At such high frequencies, policies often fail to generate actions that are both temporally smooth and spatially consistent.
We address this challenge by shifting high-frequency action learning from the action space to a latent space with variational autoencoder (VAE). This formulation significantly improves both temporal and spatial consistency of high-frequency control.
To enable smooth real-time execution, we further introduce Reuse-then-Refine, a chunk-level refine strategy that improves continuity between adjacent action chunks under asynchronous inference.
As a result, robots controlled by our policy can execute complex contact-rich tasks continuously, with less pauses and jerky motions. Experiments on three real-world contact-rich robotic tasks show that our approach consistently completes tasks with smooth motions. Our code and data are
available at https://github.com/tars-robotics/RTR.
action chunkingreal-time controllatent action spacecontact-rich manipulation
OpenReview ↗
PDF ↗
Code ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Learning Human-Robot Collaboration via Heterogeneous-Agent Lyapunov Policy Optimization
Hao Zhang, Yaru Niu, Yikai Wang, Ding Zhao, Eric H. Tseng
To improve generalization and resilience in human–robot collaboration (HRC), robots must contend with diverse combinations of human behaviors and contexts, motivating multi-agent reinforcement learning (MARL). However, inherent heterogeneity between robots and humans creates a rationality gap (RG), where decentralized policy updates deviate from cooperative joint optimization. The resulting learning problem is a general-sum differentiable game, so independent policy-gradient updates can oscillate or diverge without added structure. We propose heterogeneous-agent Lyapunov policy optimization (HALO), a framework that stabilizes decentralized MARL by enforcing Lyapunov-based contraction in policy-parameter space. Unlike Lyapunov-based safe RL, which targets state/trajectory constraints in constrained Markov decision processes, HALO uses Lyapunov certification to stabilize decentralized policy learning. HALO rectifies decentralized gradients via optimal quadratic projections, ensuring monotonic contraction of RG and enabling effective exploration of open-ended interaction spaces. Extensive simulations and real-world humanoid-robot experiments show that this certified stability improves generalization and robustness in collaborative corner cases.
human-robot collaborationmulti-agent rllyapunov stabilityhumanoid
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Learning Latent Action World Models in the Wild
Quentin Garrido, Tushar Nagarajan, Basile Terver, Nicolas Ballas, Yann LeCun, Michael Rabbat
Agents that can reason and plan in the real world must be able to predict the consequences of their actions. World models possess this capability but require action annotations that can be complex to obtain at scale. Latent action models address this issue by learning an action space from videos alone. Our work studies the training of latent action world models on in-the-wild videos, expanding the scope of existing works that focus on simple robotics simulations, video games, or manipulation data. While diverse videos enable modeling richer actions, they introduce challenges of environmental noise and lack of a common embodiment across videos. To address these, we carefully study the design and evaluation of latent actions. We find that constrained continuous latent actions are better suited for complex in-the-wild videos, compared to vector quantization. For example, actions specific to in-the-wild videos such as humans entering the room, can be modeled and then transferred across videos. However, in the absence of a common embodiment, learned latent actions are localized in space, relative to the camera. Nonetheless, we are able to train a controller that maps known actions to latent ones, allowing us to use latent actions as a universal interface to solve planning tasks on par with action-conditioned baselines.
world-modelslatent-actionsinternet-videoplanning
OpenReview ↗
PDF ↗
Poster Session 6 (Hall A) · Thursday, July 9, 10:30-12:15 Learning Manifold and Itô Dynamics with Branched Neural Rough Differential Equations
Luke Thompson, Dai Shi, Lequan Lin, Junbin Gao, Andi Han
Neural rough differential equations (NRDEs) stay accurate under irregular sampling while taking far fewer integration steps than standard neural differential equations, summarising a finely sampled driver by its log-signature and advancing the hidden state over coarse intervals using the log-ODE method. This efficiency rests on the shuffle algebra, the algebraic counterpart of Stratonovich calculus. This reliance means NRDEs cannot expose the quadratic-variation terms Itô dynamics require, nor the ordered covariant derivatives that govern Itô flows on connection-equipped manifolds. Ameliorating this, we introduce Branched Neural Rough Differential Equations (B-NRDEs), a Hopf-algebraic framework that recasts the NRDE log-ODE step as geometric numerical integration on the state-space manifold, matching the driving algebra to the governing calculus: Grossman–Larson rooted trees for Euclidean Itô dynamics, Munthe–Kaas–Wright planar rooted trees for ordered covariant derivatives on manifolds, and the shuffle algebra in the classical Stratonovich case. This yields intrinsic coarse-step dynamics that exactly preserve manifold constraints. Finally, we introduce a branched signature-kernel objective to enable Itô-consistent law matching by making quadratic-variation terms visible during training. On rough Bergomi volatility, sim-to-real $\mathrm{SO}(3)$ forecasting, and SPD covariance dynamics, B-NRDEs offer a unified, effective approach to stochastic and manifold-valued dynamics beyond the Euclidean–Stratonovich setting.
rough-differential-equationsmanifold-learningsim-to-realso3-dynamics
OpenReview ↗
PDF ↗
Code ↗
Poster Session 1 (Hall A) · Tuesday, July 7, 10:30-12:15 Learning Task-Sufficient World Models by Synergizing Agentic Exploration and Structured Modeling
Fan Feng, Yujia Zheng, Minghao Fu, Yongqiang Chen, Guangyi Chen, Kevin Patrick Murphy, Biwei Huang, Kun Zhang
Learning and planning in imagination using world models provides an effective paradigm for training agents for decision-making. However, existing approaches often rely on high-dimensional latent spaces or generic visual embeddings that retain many factors irrelevant to control, limiting efficiency and generalization across tasks. To this end, we study how agents can learn world models with representations that are task-specific, minimal, and sufficient for decision making. We achieve this via a closed-loop synergy between the agent and the world model, in which structured world-model learning distills task-sufficient representations from informative interaction data. On the agent side, agents perform active probing of the environment to collect informative trajectories that expose task-relevant latent factors, guided by an adaptive curriculum. On the world-model side, we learn structured representations over observations to distill compact, task-sufficient latent states from the collected interaction data. This synergy enables the recovery of task-sufficient latent representations that capture all control-relevant factors empirically. Leveraging these representations, the resulting policies achieve improved sample efficiency generalization, including generalization across skills, object–skill compositions, and previously unseen tasks on standard continuous control and robotic manipulation benchmarks.
world modelstask-relevant representationsexplorationmanipulation
OpenReview ↗
PDF ↗
Poster Session 1 (Hall A) · Tuesday, July 7, 10:30-12:15 Learning to Move Before Learning to Do: Task-Agnostic pretraining for VLAs
Junhao Shi, Siyin Wang, Xiaopeng Yu, Li Ji, Jingjing Gong, Xipeng Qiu
Vision-Language-Action (VLA) models are bottlenecked by the scarcity of expert demonstrations—expensive triplets of observations, language instructions, and actions. We propose that learning ''how to move'' can be decoupled from learning ''what to do,'' and that the former requires no task labels at all. Our two-stage framework, **Task-Agnostic Pretraining (TAP)** first pre-trains on abundant, cheap *task-agnostic* data (discarded off-task trajectories or autonomous robot play) using an Inverse Dynamics objective that predicts actions from consecutive observations. This self-supervised phase instills physical affordances—grasping, contact dynamics, end-effector control—without human annotation. A lightweight second stage then aligns these physical priors with language instructions using minimal expert data. On the SIMPLER benchmark, our approach matches models trained on 1M+ expert trajectories while using orders of magnitude less labeled data, achieving a 10\% absolute gain over standard behavior cloning. In real-world WidowX experiments, it surpasses internet-scale baselines under visual distribution shifts (e.g., 25\% vs. 0\% under camera perturbations), demonstrating that task-agnostic pretraining yields robust, transferable physical representations for Embodied AI.
vla pretraininginverse dynamicsself-supervised learningreal robot
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Learning to Reconfigure: Configuration-Control Co-optimization of Reconfigurable Robots for Heterogeneous Locomotion
Xiaoyu Xiong, Kehan Liu, Huiyi Yan, Shengjie Wang, Yang Gao, Tao Du
Traditional robot co-design approaches typically converge to one configuration, which do not explore the flexibility from reconfiguration on heterogeneous environments. On the other hand, existing designs for reconfigurable robots require human-designed configurations. We present Learning to Reconfigure, a holistic pipeline for configuration-control co-optimization of reconfigurable robots in heterogeneous locomotion tasks consisting of several sub-tasks. Our pipeline proposes low-level specialized primitives with a high-level scheduler. To jointly optimize configuration design and control, our primitives employ a multi-tail architecture that disentangles these distinct objectives. Building on this, the scheduler learns to dynamically switch configurations based on global task progress. We evaluate our pipeline on locomotion tasks across walking, flying, and swimming, and compare with the state-of-the-art baselines, including single-robot control and multi-morphology co-optimization algorithms. Quantitative results based on traversal progress show that our pipeline outperforms single-robot baselines by 5.95x average progress. Compared with the reconfiguration-free design given by the co-design algorithms, our robots also exhibit 9.81x progress on average. These results highlight the critical role of configuration adaptation in achieving versatile robotic autonomy in complex worlds.
reconfigurable robotsco-designlocomotionrl
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Learning Transferable Interaction Primitives from Game Videos for Humanoid Locomotion
Xiangming Zhu, Huayu Deng, Haoran Zhao, Yiwei Hao, Yunbo Wang
Learning humanoid control from video provides a scalable alternative to the scarcity of high-fidelity robot data. Existing methods, however, often rely on curated datasets and treat video as passive kinematic priors. They fail to capture dynamic humanoid interactions with the environment, which are essential for robust control in complex physical environments. To address this, we propose ***TR**ansferable **I**nteraction **P**rimitives (TRIP)*, a framework designed to extract and ground interactions from unlabeled game videos for locomotion control. TRIP explicitly models dependencies between motion dynamics and environmental context via a discrete library of interaction-based action primitives. To bridge the reality gap, we introduce a shared context latent space that aligns implicit video-domain features with functional target-domain observations, enabling the seamless transfer of video-mined strategies to reinforcement learning policies. Our experiments on complex terrain navigation demonstrate that TRIP achieves significant improvements in task performance, sample efficiency, and robustness.
humanoid locomotionsim-to-reallearning from videoterrain navigation
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 LIMMT: Less Is More for Motion Tracking
Yu Guan, Zekun Qi, Chenghuai Lin, Xuchuan Chen, Wenyao Zhang, Jilong Wang, XinQiang Yu, He Wang, Li Yi
We argue that high-quality motion data can steer tracking policies toward better optimization trajectories early in training. In this work, we introduce LIMMT (Less Is More for Motion Tracking). To our knowledge, this is the first data-centric study for physics-based humanoid motion tracking. We go beyond simply removing erroneous clips. We define motion data quality through three dimensions: physics feasibility, diversity, and complexity. We show that training with under 3% of AMASS yields better tracking performance than training with the full dataset. Extensive experiments and analyses validate the effectiveness of our framework.
motion trackinghumanoidsim-to-realdata-centric
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 LLawCo: Learning Laws of Cooperation for Modeling Embodied Multi-Agent Behavior
Qinhong Zhou, Chuang Gan, Anoop Cherian
Embodied agents operating in decentralized and partially observable environments have attracted growing attention in recent years. However, existing large language model (LLM)-based agents often exhibit behaviors that are misaligned with their partners or inconsistent with the environment state, leading to inefficient cooperation and poor task success. To address this challenge, we propose a novel framework, Learning Laws of Cooperation (LLawCo), that enables embodied agents to autonomously align with both their partners and task objectives. Our framework allows agents to reflect on past failures to extract misaligned behavioral patterns, which are used to derive high-level behavioral laws, such as “Talk when necessary” and “Wait for partner.” These laws are explicitly incorporated into the agents’ chains of thought via supervised fine-tuning, aligning their reasoning with task requirements and the behavior of other agents. To evaluate our approach, we introduce PARTNR-Dialog, a large-scale multi-agent communicative and cooperative planning benchmark built on the PARTNR environment. Experiments on existing tasks and our new benchmark demonstrate significant improvements in cooperative efficiency and task success rates. Across four backbone LLMs, our method achieves average success rate improvements of 4.5% on the PARTNR-Dialog benchmark and 6.8% on the TDW-MAT benchmark over state-of-the-art open-source communicative agent frameworks.
multi-agent cooperationembodied agentsllm fine-tuninghousehold tasks
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 Making Learner Weakness Actionable for Learning from Demonstration with Novice Teachers
Yuqing Zhu, Matthew Howard
Learning from demonstration can be an effective way to teach robots task-oriented policies. However, in an interactive setting when demonstrations are limited by time or other budgetary constraints, it is challenging to find those that fix the learner's (remaining) errors. This is especially difficult for novice teachers: they may provide task-valid trajectories, often these fail to meaningfully improve the policy due to their lack of knowledge of learning mechanisms internal to the robot. This paper introduces CLASP (Collaborative Learning with Anchored State-space Partitions), which summarises the teaching process as a compact map of behavioural regions anchored in the teacher's own demonstrations. The map connects task failure to actionable changes to demonstrations by indicating what is going wrong in an intuitive way. It also enables difficulty-aware training that emphasises regions where learning is failing. Across diverse benchmarks, CLASP improves success by up to 20\% over offline and interactive baselines under the same demonstration budget, improves robustness under distribution shift by 14–20\%, and preserves behavioural diversity.
learning from demonstrationhuman-robot interactioninteractive learning
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 ManiSoft: Towards Vision-Language Manipulation for Soft Continuum Robotics
Ziyu Wei, Luting Wang, Chen Gao, Li Wen, Si Liu
Most existing vision-language manipulation research targets rigid robotic arms, whose fixed morphology limits adaptability in cluttered or confined spaces.
Soft robotic arms offer an appealing alternative due to their deformability, but confront challenges such as unreliable proprioception and distributed low-level actuation.
To investigate these challenges, we introduce ManiSoft, a benchmark for vision-language manipulation with soft arms.
ManiSoft features a tailored simulator that couples realistic soft-body dynamics with contact-rich interactions via an elastic force constraint.
On this basis, ManiSoft defines four tasks, each highlighting distinct aspects of deformable control, from basic end-effector coordination to obstacle avoidance.
To support policy training and evaluation, ManiSoft includes an automated pipeline that generates $6{,}300$ diverse scenes and corresponding expert trajectories.
To produce high-quality trajectories at scale, we first employ a high-level planner to decompose each task into a sequence of waypoints, followed by a low-level reinforcement learning policy that generates torque commands to track waypoints.
Benchmarking three representative policy models shows relatively promising results in clean scenes but substantial performance drop under randomization.
Visualization analysis indicates that failures stem primarily from inaccurate visual estimation of proprioceptive state and limited exploitation of deformability for adaptive obstacle avoiding.
We anticipate ManiSoft to serve as a valuable testbed, bridging the gap between rigid and soft arms in the context of vision-language manipulation.
soft roboticsvision-language manipulationsimulation benchmarkdeformable objects
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 1 (Hall A) · Tuesday, July 7, 10:30-12:15 MapDream: Task-Driven Map Learning for Vision-Language Navigation
Guoxin Lian, Shuo Wang, Yucheng Wang, Yongcai Wang, Maiyue Chen, Kaihui Wang, Bo Zhang, Zhizhong Su, Deying Li, Zhaoxin Fan
Vision-Language Navigation (VLN) requires agents to follow natural language instructions in partially observed 3D environments, motivating map representations that aggregate spatial context beyond local perception.
However, most existing approaches rely on hand-crafted maps constructed independently of the navigation policy.
We argue that maps should instead be learned representations shaped directly by navigation objectives rather than exhaustive reconstructions.
Based on this insight, we propose MapDream, a map-in-the-loop framework that formulates map construction as autoregressive bird’s-eye-view (BEV) image synthesis.
The framework jointly learns map generation and action prediction, distilling environmental context into a compact three-channel BEV map that preserves only navigation-critical affordances.
Supervised pre-training bootstraps a reliable mapping-to-control interface, while the autoregressive design enables end-to-end joint optimization through reinforcement fine-tuning.
Experiments on R2R-CE and RxR-CE achieve state-of-the-art monocular performance, validating task-driven generative map learning.
vision-language navigationmap learningbev representation
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Mean Flow Policy Optimization
Xiaoyi Dong, Xi Sheryl Zhang, Jian Cheng
Diffusion models have recently emerged as expressive policy representations for online reinforcement learning (RL). However, their iterative generative processes introduce substantial training and inference overhead. To overcome this limitation, we propose to represent policies using MeanFlow models, a class of few-step flow-based generative models, to improve training and inference efficiency over diffusion-based RL approaches. To promote exploration, we optimize MeanFlow policies under the maximum entropy RL framework via soft policy iteration, and address two key challenges specific to MeanFlow policies: action likelihood evaluation and soft policy improvement. Experiments on MuJoCo, DeepMind Control Suite and HumanoidBench benchmarks demonstrate that our method, Mean Flow Policy Optimization (MFPO), achieves performance comparable to or exceeding current diffusion-based baselines while considerably reducing training and inference time. Our code is available at https://github.com/dongxiaoyi-xyz/MFPO.
online-rlflow-matchingdiffusion-policycontinuous-control
OpenReview ↗
PDF ↗
Code ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RL
Guojian Zhan, Likun Wang, Pengcheng Wang, Feihong Zhang, Jingliang Duan, Kaicheng Yu, Masayoshi Tomizuka, Shengbo Eben Li
Maximum entropy has become a mainstream off-policy reinforcement learning (RL) framework for balancing exploitation and exploration. However, two bottlenecks still limit further performance gains: (1) non-stationary Q-value estimation stemming from the joint injection of entropy and the concurrent updating of its temperature parameter; and (2) short-sighted local entropy tuning, which adjusts temperature solely based on current single-step entropy without accounting for cumulative entropy over time. In this paper, we broaden the maximum entropy framework by proposing a trajectory entropy-constrained reinforcement learning (TECRL) framework to address these limitations. We begin by introducing reward-entropy separation (RES) to decouple the value targets, ensuring they remain stable and unaffected by temperature fluctuations. Subsequently, the resulting entropy Q-function is leveraged to explicitly quantify expected cumulative entropy, allowing for the enforcement of a trajectory entropy constraint (TEC) to govern long-term stochasticity. We instantiate this framework as DSAC-E, a practical off-policy algorithm that builds upon the latest distributional soft actor-critic. Extensive evaluations across 10 challenging tasks in locomotion, robotic manipulation, and vision-based driving domains demonstrate that DSAC-E consistently outperforms baselines in both cumulative returns and training stability.
maximum entropy rlrobotic manipulationlocomotionactor-critic
OpenReview ↗
PDF ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 MindZero: Learning Online Mental Reasoning With Zero Annotations
Shunchi Zhang, Jin Lu, Chuanyang Jin, Yichao Zhou, Zhining Zhang, Tianmin Shu
Effective real-world assistance requires AI agents with robust Theory of Mind (ToM): inferring human mental states from their behavior. Despite recent advances, several key challenges remain, including (1) online inference with robust uncertainty updates over multiple hypotheses; (2) efficient reasoning suitable for real-time assistance; and (3) the lack of ground-truth mental state annotations in real-world domains. We address these challenges by introducing *MindZero*, a self-supervised reinforcement learning framework that trains multimodal large language models (MLLMs) for efficient and robust online mental reasoning. During training, the model is rewarded for generating mental state hypotheses that maximize the likelihood of observed actions estimated by a planner, similar to model-based ToM reasoning. This method thus eliminates the need for explicit mental state annotations. After training, *MindZero* internalizes model-based reasoning into fast single-pass inference. We evaluate *MindZero* against baselines across challenging mental reasoning and AI assistance tasks in gridworld and household domains. We found that LLMs alone are insufficient; model-based methods improve accuracy but are slow, costly, and limited by backbone MLLM capacity. In contrast, *MindZero* enhances MLLMs' intrinsic ToM ability and significantly outperforms model-based methods in both accuracy and efficiency, showing that mental reasoning can be effectively learned as a self-supervised skill.
theory of mindhousehold simulationembodied agentsself-supervised rl
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 1 (Hall A) · Tuesday, July 7, 10:30-12:15 Mitigating Error Accumulation in Continuous Navigation via Memory-Augmented Kalman Filtering
Yin Tang, Jiawei Ma, Jinrui Zhang, Alex Jinpeng Wang, Deyu Zhang
Continuous prediction in complex environments is critical for Unmanned Aerial Vehicle (UAV). However, the existing Vision-Language Navigation (VLN) models follows the dead-reckoning, which iteratively predicts the next waypoint and updates its position, thereby constructing the complete trajectory. Then, such stepwise manner will inevitably lead to accumulated errors of position over time, resulting in misalignment between internal belief and objective coordinates, which is known as ``state drift'' and ultimately compromises the subsequent trajectory prediction. Drawing inspiration from classical control theory, we propose to correct for errors by formulating the continuous prediction as a recursive Bayesian state estimation problem. In this paper, we design NeuroKalman, a novel framework that decouples navigation into two complementary processes: a Prior Prediction, based on motion dynamic,s and a Likelihood Correction, from historical observation. We first mathematically associate Kernel Density Estimation of the measurement likelihood with the attention-based retrieval mechanism, which then allows the system to rectify the latent representation using retrieved historical anchors without gradient updates. Comprehensive experiments on TravelUAV benchmark demonstrate that, with only 10\% of the full training data fine-tuning, our method clearly outperforms strong baselines and regulates drift accumulation.
uavvision-language navigationkalman filteringdrift correction
OpenReview ↗
PDF ↗
Poster Session 6 (Hall A) · Thursday, July 9, 10:30-12:15 Mixture of Horizons in Action Chunking
Dong Jing, Gang Wang, Jiaqi Liu, Weiliang Tang, Zelong Sun, Yunchao Yao, Zhenyu Wei, Yunhui Liu, Zhiwu Lu, Mingyu Ding
Vision-language-action models exhibit an inherent trade-off in action chunk length (``horizon''): longer horizons improve global foresight but degrade fine-grained local control, while shorter ones yield the opposite.
To mitigate the trade-off, we propose a $\textbf{mixture of horizons (MoH)}$ strategy.
In brief, MoH rearranges the action chunk into several segments with different horizons, processes them in parallel with a shared action transformer, and fuses outputs with a light linear gate.
It offers three appealing benefits.
i) Long-term foresight and short-term precision are jointly exploited within a single model.
ii) MoH is plug-and-play for full-attention action modules with minimal training or inference overhead.
iii) MoH enables dynamic inference with adaptive horizons, which selects stable actions through cross-horizon consensus, achieving 2.5$\times$ higher throughput than baselines while preserving superior performance.
Extensive experiments over flow-based and one-step regression policies demonstrate that MoH yields consistent and significant gains on both simulations and real-world tasks.
vision-language-actionaction chunkingrobot manipulationreal-world deployment
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Model-Based Diffusion Sampling for Predictive Control in Offline Decision Making
Haldun Balim, Na Li, Yilun Du
Offline decision-making via diffusion models often produces trajectories that are misaligned with system dynamics, limiting their reliability for control. We propose *Model Predictive Diffuser* (MPDiffuser), a compositional diffusion framework that combines a diffusion planner with a dynamics diffusion model to generate task-aligned and dynamically plausible trajectories. MPDiffuser interleaves planner and dynamics updates during sampling, progressively correcting feasibility while preserving task intent. A lightweight ranking module then selects trajectories that best satisfy task objectives. The compositional design improves sample efficiency and adaptability by enabling the dynamics model to leverage diverse and previously unseen data independently of the planner. Empirically, we demonstrate consistent improvements over prior diffusion-based methods on unconstrained (D4RL) and constrained (DSRL) benchmarks, and validate practicality through deployment on a real quadrupedal robot.
diffusion planningmodel predictive controlquadrupedoffline rl
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Moment Matching Q-Learning
Yiyan Liang Edgar, Sifei Liu, Weitong Zhang
Score-based and flow-based generative models exhibit remarkable expressive capacity in capturing complex distributions, and have been extensively deployed in tasks ranging from image generation to reinforcement learning.
Nevertheless, these models suffer from prolonged inference latency, which imposes a significant computational bottleneck in RL with iterative sampling.
To overcome this limitation, we propose a new framework named *Moment Matching Q-Learning* (MoMa QL), which utilizes a technique from statistical hypothesis testing known
as maximum mean discrepancy (MMD) that intend to match all orders of statistics between the original and target distribution.
By enforcing strong regularization on all moment statistics, this algorithm guarantees distribution-level convergence for conditional score function and remains stable under various hyperparameters.
Empirically, we show that our method MoMa QL is more computationally efficient with a comparable if not competitive performance in various D4RL tasks. Remarkably, by accelerating the action sampling process for flow-based policies, MoMa QL demonstrates superior performance in offline-to-online RL tasks because of faster and stronger adaptability for online interactive finetuning.
q-learningoffline-rlgenerative-modelsd4rl
MoSA: Motion-constrained Stress Adaptation for Mitigating Real-to-Sim Gap in Continuum Dynamics via Learning Residual Anisotropy
Jiaxu Wang, Junhao He, Jingkai SUN, Yi Gu, Yunyang Mo, Jiahang Cao, Qiang Zhang, Renjing Xu
Learning real-world dynamics from visual observations is crucial for various domains. A common strategy is to calibrate simulators by estimating physical parameters, yet accuracy is ultimately bounded by the underlying physical models, which often assume materials are homogeneous and isotropic. Even if reasonable, real-world objects typically exhibit mild anisotropy and heterogeneity. After the near-isotropic backbone is well calibrated, these residual effects become the key bottleneck for further closing the real-to-sim gap. Although neural networks can fit dynamics end-to-end, such black-box modeling discards strong physical priors, leading to poor data efficiency and overfitting.
Therefore, we propose MoSA, a motion-constrained stress adaptation framework that targets these residual effects to further improve real-to-sim dynamics learning. MoSA uses an isotropic model as a physics prior and learns residual stress operators to capture mild anisotropy and heterogeneity. It progressively adapts stresses via microplane-constrained redistribution in a physics-informed cascaded network. We further impose motion constraints by supervising temporal and spatial derivatives of the deformation field. Experimentally, our learned dynamics achieves superior accuracy, generalization, and robustness, while learning physically meaningful residual anisotropy. Finally, we validate MoSA in a robot manipulation setting, showing that better real-to-sim dynamics modeling translates into more reliable sim-to-real transfer.
real-to-simdeformable object manipulationphysics-informedsim-to-real transfer
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Mosaic: Runtime-Efficient Multi-Agent Embodied Planning
Kunjal Panchal, Saayan Mitra, Sunav Choudhary, Victor Bursztyn, Somdeb Sarkhel, Hui Guan
LLM-based multi-agent embodied planning remains impractical due to prohibitively high execution latency. We identify failed actions as the dominant bottleneck, stemming from two core challenges: inaccurate state tracking under partial observability and inefficient coordination that produces redundant or conflicting actions. We introduce Mosaic, a runtime-efficient multi-agent planning framework that addresses both challenges. Mosaic maintains accurate yet lightweight state tracking through agent-centric semantic memory that stores objects in relative coordinates, enabling geometric transformations and coordination. It ensures efficient coordination through Integer Linear Programming that allocates actions at every planning step, enforcing physical feasibility and inter-agent coordination constraints. Across AI2-THOR and search-and-rescue benchmarks, Mosaic achieves 27–32% faster execution, 30–33% fewer LLM calls, 25–31% fewer steps, and 4–10% points higher success rates. These results demonstrate that efficient memory and constraint-guided coordination are critical for scalable, low-latency multi-agent planning.
multi-agent planningai2-thorembodied planningcoordination
Motion Dynamics Learning for Few-Shot Embodied Adaptation
Sibo He, Weiying Xie, Daixun Li, Junhao Zhong, Jiayun Tian, Yunke Wang, Leyuan Fang, Gang He, Yunsong Li
Vision-Language-Action (VLA) models have shown strong potential for robotic manipulation, yet adapting pretrained models to novel tasks typically relies on substantial task-specific demonstrations, limiting scalability.
Current VLA methods mostly focus on action imitation, which ignores the richer structure contained in trajectories. In contrast, motion dynamics governing how actions evolve over time are more informative and transferable, making them better suited for few-shot adaptation.
Motivated by this idea, we propose DynVLA, a few-shot adaptation system that reformulates VLA learning from action imitation to trajectory-level motion dynamics modeling.
Specifically, we propose Motion Dynamics Mechanism (MDM), which distills latent embeddings from trajectories via flow-matching inversion, yielding compact representations that capture dynamics.
We further design Dynamics-Constrained Modeling (DCM). DCM projects these inferred representations onto a Dynamics Bank, which stores prior motion knowledge pretrained from diverse demonstrations. By grounding action generation in these learned priors, the system enables interpolating between existing action paradigms to represent novel dynamics modes.
Experiments on 13 real-world tasks demonstrate that DynVLA outperforms existing SOTA systems by 15.8\% in average success rate with around 20 demonstrations, highlighting its adaptation capabilities in real-world scenes. Our code is available at https://github.com/trantor2nd/Motion-DynVLA.
vision-language-actionfew-shot adaptationrobotic manipulationmotion dynamics
OpenReview ↗
PDF ↗
Code ↗
Poster Session 6 (Hall A) · Thursday, July 9, 10:30-12:15 Motion Planning in Compressed Representation Spaces
Lukas Lao Beyer, Sertac Karaman
Deep learning methods have vastly expanded the capabilities of motion planning in robotics applications, as learning priors from large-scale data has been shown to be essential in capturing the highly complex behavior required for solving tasks such as manipulation or navigation for autonomous vehicles. At the same time, model-based planning algorithms based on search or optimization remain an essential tool due to their flexibility, efficiency, and the ability to incorporate domain knowledge via expert-designed algorithms and objective functions. We propose a new generative framework to unify these two paradigms. First, we learn an autoencoder with a high compression ratio and a latent space of hierarchically ordered, discrete-valued tokens. Leveraging both the dimensionality reduction and the hierarchical coarse-to-fine structure learned by this autoencoder, we then perform motion planning by directly searching in the latent space of tokens. This search can optimize arbitrary objective functions specified at test time, providing a large degree of flexibility while maintaining efficiency and producing realistic solutions by relying on the generative capabilities of the highly compressed autoencoder. We evaluate our method on nuPlan and the Waymo Open Motion Dataset, showing how latent space search can be used for a variety of guided behavior generation tasks, achieving strong performance for closed-loop motion planning and multi-agent guided scenario synthesis without requiring any task-specific training.
motion planningautonomous vehicleslatent space searchtokenization
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Move-Then-Operate: Behavioral Phasing for Human-Like Robotic Manipulation
Haoming Xu, Lei Lei, Jie Gu, Chu Tang, Jingmin Chen, Rui-Qi Wang
We present Move-Then-Operate, a Vision–language–action framework that explicitly decouples robotic manipulation into two distinct behavioral phases: coarse relocation (move) and contact-critical interaction (operate). Unlike monolithic policies that conflate these heterogeneous regimes, our architecture employs a dual-expert policy routed by a learnable phase selector, introducing a structural inductive bias that isolates phase-specific dynamics. Phase labels are automatically generated via an MLLM-based pipeline conditioned on lightweight contextual cues such as end-effector velocity and subtask decomposition to ensure alignment with human motor patterns. Evaluated on the RoboTwin2 benchmark, our method achieves an average success rate of $68.9\%$, outperforming the monolithic $\pi_0$ baseline by +$24\%$. It matches or exceeds models trained on $10\times$ more data and reaches peak performance in $40\%$ fewer training steps, demonstrating that architectural disentanglement of move and operate phases is a highly effective and efficient strategy for mastering high-precision manipulation.
vision-language-actiondual-expert policyrobotic manipulation
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Moving Out: Physically-grounded Human-AI Collaboration
Xuhui Kang, Sung-Wook Lee, Haolin Liu, Yuyan Wang, Yen-Ling Kuo
The ability to adapt to physical actions and constraints in an environment is crucial for embodied agents (e.g., robots) to effectively collaborate with humans.
Such physically grounded human-AI collaboration must account for the increased complexity of the continuous state-action space and constrained dynamics caused by physical constraints.
However, most existing collaboration benchmarks are discrete or do not consider physical attributes and constraints.
To address this, we introduce Moving Out, a human-AI collaboration benchmark that resembles a wide range of collaboration modes affected by physical attributes and constraints, such as moving heavy items together and coordinating actions to move an item around a corner.
Moving Out consists of two challenges and human-human interaction data to comprehensively evaluate models' abilities to adapt to diverse human behaviors and unseen physical attributes.
To give embodied agents the capability to collaborate with humans under physical attributes and constraints, we propose a novel method, BASS (Behavior Augmentation, Simulation, and Selection), to enhance the diversity of agents and their understanding of the outcome of actions.
We systematically compare BASS and state-of-the-art models in AI-AI and human-AI experiments, showing that BASS can effectively collaborate with both unseen AI and humans.
human-ai collaborationphysical constraintsbenchmarkembodied agents
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 MSP: Probabilistically Consistent Multi-Scale Action Generation
Zhixuan Lin, Gengqi Liu, Chao Zheng, Gao Lin, Jindong Yu, Song Gao, Fei Wang
In robotic imitation learning, accurately modeling the multimodality and temporal correlations of long-horizon action sequences remains challenging. Long-horizon tasks require preserving global task intent while executing precise low-level control; otherwise, local errors can accumulate and lead to failure. While recent coarse-to-fine autoregressive models have improved action generation, they struggle to maintain consistency across hierarchies, leading to suboptimal performance in long-horizon tasks. To address these shortcomings, we propose Probabilistically Consistent Multi-Scale Action Generation (MSP), a novel coarse-to-fine approach that promotes cross-scale consistency. MSP adopts a streamlined multi-scale design by directly downsampling in a continuous latent space. A scale-wise autoregressive Transformer is used to generate semantic conditions at each scale, which guide a lightweight MeanFlow model to capture multi-scale latent distributions, enabling probabilistically consistent refinement across scales. Through extensive simulation and real-world experiments, including long-horizon, multi-task, and few-shot generalization settings, we show that MSP outperforms existing coarse-to-fine methods, achieving state-of-the-art performance with high efficiency.
robotic imitation learninglong-horizon manipulationaction generationmulti-scale policy
Multi-Label Learning with Contrastive Cluster Self-Supervision for 3D Hierarchical Semantic Segmentation
Shuyu Cao, Chongshou Li, Jie Xu, Tianrui Li, Na Zhao
3D hierarchical semantic segmentation (3DHS) is crucial for embodied intelligence that demands the coarse-to-fine grained and multi-hierarchy understanding of 3D scenes. 3DHS tasks can be addressed by multi-label learning, but facing two issues: I) learning multiple labels for each point with a shared model can lead to multi-hierarchy conflicts in cross-hierarchy optimization, and II) the class imbalance issue is inevitable across multiple hierarchies of 3D scenes, making the model easily be dominated by major classes. To address these issues, we propose a novel multi-label learning with contrastive cluster self-supervision framework for 3DHS. Specifically, we propose a late-decoupled multi-label learning 3DHS network which employs decoupled decoders with the coarse-to-fine hierarchical consistency guidance. This late-decoupled model architecture can mitigate the underfitting and overfitting conflicts among multiple hierarchies and also constrain the class imbalance problem within each individual hierarchy. Moreover, we introduce a 3DHS-oriented contrastive cluster self-supervision learning method, which learns cluster-wise point cloud features with contrastive loss and produces self-supervised information to enhance the class-imbalance segmentation. Extensive experiments on multiple datasets and backbones demonstrate that our approach promotes the multi-hierarchy balance and mitigates the class imbalance issue in 3DHS tasks.
3d-segmentationpoint-cloudself-supervisedscene-understanding
OpenReview ↗
PDF ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 Multi-Objective Learning for Diffusion Models: A Statistical Theory under Semi-Supervised Learning
Ziheng Cheng, Yixiao Huang, Hanlin Zhu, Haoran Geng, Somayeh Sojoudi, Jitendra Malik, Pieter Abbeel, Xin Guo
Diffusion models are increasingly used as powerful conditional generators, yet real deployments often involve multiple target distributions arising from different tasks, e.g., diverse prompt domains in text-to-image generation, or multiple environments in robotics with diffusion policies.
This naturally leads to a multi-objective learning (MOL) problem.
A key challenge is that achieving good Pareto trade-offs can require a generalist model class with substantially larger capacity than what suffices for solving any individual task, thereby increasing statistical cost since sample complexity typically scales with the model complexity.
To reconcile this, we develop a principled MOL framework for diffusion models with limited data: a semi-supervised regime where paired (labeled) samples are scarce, but (unlabeled) condition data are abundant.
We propose a two-stage training procedure that first fits lightweight specialist models from limited paired data, and then distills them into a generalist model by generating pseudo-samples.
We establish generalization bounds showing that the required number of paired samples only depends on the complexity of the specialist model classes.
We further extend the theory to diffusion policies for sequential decision making to account for distribution shift in on-policy rollouts.
Extensive experiments on robotic control and image restoration tasks are conducted to verify our theoretical results.
diffusion-modelsmulti-objective-learningstatistical-theorydiffusion-policy
Must All Negatives Be Pushed Away Equally? Uncertainty-Aware Cross-View Geo-Localization via Normal Inverse Gamma Distribution
Songsong Ouyang, Le Wu, Yingying Zhu
Cross-view geo-localization (CVGL) aims to retrieve the corresponding satellite image given a street query and is critical for autonomous navigation. Although recent methods perform well on benchmarks, they often fail to generalize to unseen environments. A key limitation is the use of contrastive learning, which assigns equal labels to all negative samples and induces similarity-amplified repulsion. But should all negatives be treated equally? In CVGL, semi-positive samples that are geographically proximate to the positive often share important semantic cues. Treating them as ordinary negatives forces the model to overfit noise, leading to a collapse in generalization. To address this issue, we propose an uncertainty-aware framework grounded in Deep Evidential Regression (DER), modeling the Normal-Inverse-Gamma (NIG) distribution as a conjugate prior to quantify environmental complexity $u$ in a single forward pass. The estimated $u$ adaptively softens labels for hard negatives in Soft InfoNCE, mitigating excessive repulsion on semi-positive samples. An Uncertainty Head with cls-to-spatial cross-attention and attention statistics is designed to accurately fit the NIG distribution. Extensive experiments demonstrate state-of-the-art performance, including an average 18\% R@1 improvement in zero-shot cross-dataset transfer, filling the critical gap between laboratory benchmarks and robust real-world deployment.
geo-localizationcross-view-retrievaluncertainty-estimationautonomous-navigation
MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulation
Jiaxu Wang, JIANG Yicheng, Tianlun HE, Jingkai SUN, Qiang Zhang, Jiahang Cao, Zesen Gan, Mingyuan Sun, Qiming Shao, Xiangyu Yue
World-model-based imagine-then-act becomes a promising paradigm for robotic manipulation, yet existing approaches typically support either purely image-based forecasting or reasoning over partial 3D geometry, limiting their ability to predict complete 4D scene dynamics. To solve this, this work explores a novel embodied 4D world model that enables geometrically consistent, arbitrary-view RGBD generation: given only a single-view RGBD observation as input, the model “imagines” the remaining viewpoints, which can then be back-projected and fused to assemble a more complete 3D structure across time. To efficiently learn the multi-view, cross-modality generation, we explicitly design cross-view and cross-modality feature fusion that jointly encourage consistency between RGB and depth and enforce geometric alignment across views. Beyond prediction, converting generated futures into actions is often handled by inverse dynamics, which is ill-posed because multiple actions can explain the same transition. We address this with a test-time action optimization strategy that backpropagates through the generative model to infer a trajectory-level latent best matching the predicted future, and a residual inverse dynamics model that turns this trajectory prior into accurate executable actions. Extensive experiments on the three datasets and platforms demonstrate strong performance on both 4D scene generation and downstream manipulation, and ablations provide practical insights into the key design choices. Project page is available at \url{https://mercerai.github.io/MVISTA-4D/}.
world modelrobot manipulation4d scene generationtest-time action inference
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstruction
Jung Min Lee, Dohyeok Lee, Seokhun Ju, Taehyun Cho, Jin Woo Koo, Li Zhao, Sangwoo Hong, Jungwoo Lee
Latent actions learned from diverse human videos serve as pseudo-labels for vision-language-action (VLA) pretraining, but provide effective supervision only if they remain informative about the underlying ground-truth actions. For effective supervision, latent actions should contain information about the underlying actions even though they are inaccessible. We propose **M**ulti-**V**iew**P**oint **L**atent **A**ction **M**odel (**MVP-LAM**), which learns latent actions that are highly informative about ground-truth actions from multi-view videos. MVP-LAM trains latent actions with a *cross-viewpoint reconstruction* objective, so that a latent action from one view must explain the future in another view, reducing reliance on viewpoint-specific cues. On Bridge V2, MVP-LAM produces more action-centric latent actions, achieving higher mutual information with ground-truth actions and improved action prediction, including under out-of-distribution evaluation. Finally, pretraining VLAs with MVP-LAM latent actions improves downstream manipulation performance on various benchmarks. The code and trained checkpoints are available at [https://jm-this.github.io/mvp_lam/](https://jm-this.github.io/mvp_lam/).
latent action modelvla pretrainingcross-view reconstructionmanipulation
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 N2M: Bridging Navigation and Manipulation by Learning Pose Preference from Rollout
Kaixin Chai, Hyunjun Lee, Joseph J Lim
Determining where to execute the manipulation policy is a fundamental challenge in mobile manipulation. Most approaches have formulated this as a geometric search problem, prioritizing physical reachability. However, given the high sensitivity of modern learning-based manipulation policies, geometric criteria alone are insufficient. Optimal performance requires base positioning that is aware of the policy's preference. While recent works have attempted to address this, they remain limited in practicality due to reliance on pre-built scene reconstruction and slow inference. In this work, we introduce **N2M** that systematically reformulates the approach to base positioning problem, naturally overcoming limitations of previous methods. Our key insight is that policy preferences are inherent to the local scene structure and can be effectively learned from the policy rollouts. Technically, we propose a novel *viewpoint augmentation* strategy that enables the model to learn robust, viewpoint-invariant pose preferences with remarkable data efficiency. Extensive experiments demonstrate that N2M achieves state-of-the-art performance, outperforming both non-policy-aware baselines and recent policy-aware alternatives. Furthermore, we provide a comprehensive analysis highlighting N2M’s broad applicability, generalization capabilities, and data efficiency. Project website: https://clvrai.github.io/N2M/
mobile manipulationbase positioningpolicy-aware placementviewpoint augmentation
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 NavOL: Navigation Policy with Online Imitation Learning
Xiaofei Wei, Chun Gu, Li Zhang
Learning robust navigation policies remains a core challenge in robotics. Offline imitation learning suffers from distribution shift and compounding errors at rollout, while reinforcement learning requires reward engineering and learns inefficiently. In this paper, we propose NavOL, an online imitation learning paradigm that interacts with a simulator and updates itself using expert demonstrations gathered online. Built upon a pretrained navigation diffusion policy that maps local observations to future waypoints, NavOL trains in a rollout–update loop: during rollout, the policy acts in the simulator and queries a global planner which has privileged access to the global environment for the optimal path segment as ground truth trajectory labels; during update, the policy is trained on the online collected observation–trajectory pairs. This online imitation loop removes the need for reward design, improves learning efficiency, and mitigates distribution shift by training on the policy’s own explored rollouts. Built on IsaacLab with fast, high-fidelity parallel rendering and domain randomization of camera pose and start-goal pairs, our system scales across 50 scenes on 8 RTX 4090 GPUs, collecting over 2,000 new trajectories per hour, each averaging more than 400 steps. We also introduce an indoor visual navigation benchmark with predefined start and goal positions for zero-shot generalization. Extensive evaluations on simulation benchmarks, including the NavDP benchmark and our proposed benchmark, as well as carefully designed real-world experiments, demonstrate the effectiveness of NavOL, showing consistent performance gains in online imitation learning.
navigation policyimitation learningdiffusion policysim-to-real
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Neural Control: Adjoint Learning Through Equilibrium Constraints
Dezhong Tong, Jiawen Wang, Hengyi Zhou, Yinlong Shen, Xiaonan Huang, M. Khalid Jawed
Many physical AI tasks require sequential implicit computation: at each step, boundary controls are applied, and the resulting configuration is obtained by solving an equilibrium problem. This setting arises naturally in deformable object manipulation, where even bending a deformable linear object (DLO) to a target shape can be nonlinear and multistable: identical boundary conditions may produce different configurations depending on actuation history. Unlike explicit transition models, the control-to-configuration relation is implicit and history-dependent, making long-horizon learning and control brittle; backpropagating through iterative solves is also memory- and compute-intensive. We propose Neural Control, a boundary-control framework that propagates gradients through branch-dependent sequences of equilibrium solves rather than a single fixed point. Neural Control computes trajectory-dependent proxy gradients by differentiating equilibrium conditions with an adjoint formulation, avoiding solver unrolling while keeping forward rollouts on converged equilibria. Combined with receding-horizon continuation, Neural Control re-anchors optimization to realized equilibria and mitigates basin switching. We validate Neural Control on simulated and real DLO manipulation, compare against SPSA and iCEM, and demonstrate applicability to a learned DEQ-style implicit equilibrium model.
deformable object manipulationdifferentiable physicsrobot learningadjoint method
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Neural Implicit Action Fields: From Discrete Waypoints to Continuous Functions for Vision-Language-Action Models
Haoyun Liu, Jianzhuang Zhao, Xinyuan Chang, Tianle Shi, Chuanzhang Meng, Jiayuan Tan, Feng Xiong, Tong Lin, Dongjie Huo, Mu Xu, SongLin Dong, Zhiheng Ma, Yihong Gong, Sheng Zhong
Despite the rapid progress of vision-language-action (VLA) models, the prevailing practice of predicting action chunks as discrete waypoints remains structurally misaligned with the intrinsic continuity of physical motion. This discretization arises naturally from fixed-rate robot data collection and the token-by-token prediction paradigm of large language models, but ties actions to rigid sampling rates, does not naturally support analytically consistent higher-order derivatives, and introduces quantization artifacts that hinder precise, compliant interaction. We propose Neural Implicit Action Fields (NIAF), which reformulates chunk-level action representation from discrete waypoints to continuous action functions. Using a vision-language model as a hierarchical spectral modulator over a learnable motion prior, NIAF synthesizes continuous-time action manifolds with arbitrary temporal resolution. This formulation enables analytical differentiation, allowing explicit supervision of velocity and regularization of higher-order derivative signals to promote mathematical consistency, physical plausibility, and control smoothness. Our approach achieves strong results on CALVIN and LIBERO across diverse backbones. Real-world experiments further confirm that NIAF supports stable impedance control, bridging policy-side action generation and execution-side smooth control.
vision-language-actioncontinuous action representationimpedance controlrobot manipulation
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Neuro-evolutionary Continual Reinforcement Learning
Pengyi Li, Hongyao Tang, Yifu Yuan, YAN ZHENG, Xin Xu, Jianye HAO
Deploying robots in open-ended real-world environments demands continual learning capabilities to adapt to an ever-expanding range of tasks. This requires retaining previously acquired skills without forgetting while effectively leveraging prior knowledge to learn new ones. Inspired by neuroscience, we propose **N**euro-**e**volutionary **C**ontinual **R**einforcement **L**earning (**Nevo-CRL**). Nevo-CRL maintains a fixed-capacity monolithic policy network, solving tasks by optimizing inter-layer connectivity and neuron parameters.
For each new task, Nevo-CRL constructs a mask population to selectively activate the outputs of each hidden layer, thereby forming a task-specific policy population. Upon completing each task, the best-performing mask is stored, and its activated neurons are frozen to prevent catastrophic forgetting. To facilitate knowledge transfer, Nevo-CRL reuses neurons from acquired skills based on semantic similarity between tasks, while dynamically allocating additional neurons for task-specific adaptation.
In the learning process, Nevo-CRL iteratively adjusts masks via importance-guided crossover to optimize the policy network connectivity. To improve neuron utilization, we prune low-activity connections to recycle neurons. Experiments demonstrate that Nevo-CRL achieves state-of-the-art performance among continual RL methods.
The code is available at [https://github.com/yeshenpy/Nevo-CRL](https://github.com/yeshenpy/Nevo-CRL).
continual-learningneuroevolutionreinforcement-learningcatastrophic-forgetting
OpenReview ↗
PDF ↗
Code ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 NeurVLA: Unleashing Failure-Handling Capability of Vision-Language-Action Models via Neural-Symbolic Reasoning
Xuqi Liu, Minghe Gao, Juncheng Li, Siliang Tang
Vision-Language-Action models have recently shown promising progress in embodied robotic manipulation, yet their generalization to diverse open-ended embodied tasks is often hindered by execution failures. While prior work has explored failure handling, existing approaches still suffer from two fundamental limitations: coarse-grained failure correction and unreliable failure prevention. These limitations lead to brittle decision-making when VLA models are deployed in novel tasks and environments. To address them, we propose NeurVLA, a neural-symbolic framework that jointly addresses failure correction and prevention via neural-symbolic reasoning and further internalizes these failure-handling capabilities into VLA models. Experiments demonstrate that NeurVLA achieves strong performance and robust generalization across diverse tasks.
vision-language-actionfailure handlingneural-symbolic reasoningrobotic manipulation
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 NNiT: Width-Agnostic Neural Network Generation with Structurally Aligned Weight Spaces
Jiwoo Kim, Swarajh Mehta, Hao-Lun Hsu, Hyunwoo Ryu, Yudong Liu, Miroslav Pajic
Generative modeling of neural network parameters is often tied to architectures because standard parameter representations rely on known weight-matrix dimensions. Generation is further complicated by permutation symmetries that allow networks to model similar input-output functions while having widely different, unaligned parameterizations. In this work, we introduce Neural Network Diffusion Transformers (NNiTs), which generate weights in a width-agnostic manner by tokenizing weight matrices into patches and modeling them as locally structured fields. We establish that Graph HyperNetworks (GHNs) with a convolutional neural network (CNN) decoder structurally align the weight space, creating the local correlation necessary for patch-based processing. Focusing on Multilayer Perceptrons (MLPs), where permutation symmetry is especially apparent, NNiTs generate fully functional networks across a range of architectures. Our approach jointly models discrete architecture tokens and continuous weight patches within a single sequence model. On ManiSkill3 robotics tasks, NNiT achieves $>85\%$ success on architecture topologies unseen during training, while baseline approaches fail to generalize; the same pipeline also generalizes to MNIST classification beyond the robotic control setting.
weight-space-learningneural-network-diffusiongenerative-modelingmaniskill
OpenReview ↗
PDF ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 Noise-Guided Transport: Imitation Learning from Random Priors
Lionel Blondé, Joao Candido Ramos, Alexandros Kalousis
We consider imitation learning in the low-data regime, where only a limited number of expert demonstrations are available. In this setting, methods that rely on large-scale pretraining or high-capacity architectures can be difficult to apply, and efficiency with respect to demonstration data becomes critical. We introduce Noise-Guided Transport (NGT), a lightweight off-policy method that casts imitation as an optimal transport problem solved via adversarial training. NGT requires no pretraining or specialized architectures, incorporates uncertainty estimation by design, and is easy to implement and tune. Despite its simplicity, NGT achieves strong performance on challenging continuous control tasks, including high-dimensional Humanoid tasks, under ultra-low data regimes with as few as 20 transitions.
imitation-learningoptimal-transportlow-datahumanoid
OpenReview ↗
PDF ↗
Poster Session 8 (Hall A) · Thursday, July 9, 5:00-6:45 OGPO: Sample Efficient Full-Finetuning of Generative Control Policies
Sarvesh Patil, Mitsuhiko Nakamoto, Manan Agarwal, Shashwat Saxena, Jesse Zhang, Giri Anantharaman, Cleah Winston, Chaoyi Pan, Douglas Chen, Nai-Chieh Huang, Zeynep Temel, Oliver Kroemer, Sergey Levine, Abhishek Gupta, Hongkai Dai, Paarth Shah, Max Simchowitz
Generative control policies (GCPs), such as diffusion- and flow-based control policies, have proved effective parameterizations for robot learning. This work introduces Off-policy Generative Policy Optimization (**OGPO**), a sample-efficient algorithm for finetuning GCPs that maintains off-policy critics to maximize data reuse and propagate policy gradients through the full generative process of the policy via a modified PPO objective, using critics as the terminal reward. **OGPO** achieves state-of-the-art performance on manipulation tasks spanning multi-task settings, high-precision insertion, and dexterous control. To our knowledge, it is also the only method that can *fine-tune poorly-initialized behavior cloning policies to near full task-success with no expert data in the online replay buffer*, and does so with *few task-specific hyperparameter tuning*. Through extensive investigations, we demonstrate that **OGPO** drastically outperforms alternative methods on policy steering and learning residual corrections, and identify the key mechanisms behind its performance. We further introduce practical stabilization tricks, including success-buffer regularization and two-sided conservative advantages to mitigate critic over-exploitation across state- and pixel-based settings. Beyond proposing **OGPO**, we conduct a systematic empirical study of GCP finetuning, identifying the stabilizing mechanisms and failure modes that govern successful off-policy full-policy improvement.
generative control policyrl finetuningdexterous manipulationoff-policy rl
OpenReview ↗
PDF ↗
Code ↗
Poster Session 7 (Hall A) · Thursday, July 9, 2:30-4:15 OMP: One-step Meanflow Policy with Directional Alignment
Han Fang, Yize Huang, Yuheng Zhao, Paul Weng, Xiao Li, Yutong Ban
Robot manipulation has increasingly adopted data-driven generative policy frameworks, yet the field faces a persistent trade-off: diffusion models suffer from high inference latency, while flow-based methods often require complex architectural constraints. Although in image generation domain, the MeanFlow paradigm offers a path to single-step inference, its direct application to robotics is impeded by critical theoretical pathologies, specifically spectral bias and gradient starvation in low-velocity regimes. To overcome these limitations, we propose the One-step MeanFlow Policy (OMP), a novel framework designed for high-fidelity, real-time manipulation. We introduce a lightweight directional alignment mechanism to explicitly synchronize predicted velocities with true mean velocities. Furthermore, we implement a Differential Derivation Equation (DDE) to approximate the Jacobian-Vector Product (JVP) operator, which decouples forward and backward passes to significantly reduce memory complexity. Extensive experiments on the Adroit and Meta-World benchmarks demonstrate that OMP outperforms state-of-the-art methods in success rate and trajectory accuracy, particularly in high-precision tasks, while retaining the efficiency of single-step generation.
meanflowimitation learningone-step policyrobot manipulation
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 8 (Hall A) · Thursday, July 9, 5:00-6:45 On the Sample Efficiency of Inverse Dynamics Models for Semi-Supervised Imitation Learning
Sacha Morin, Moonsub Byeon, Alexia Jolicoeur-Martineau, Sebastien Lachapelle
Semi-supervised imitation learning (SSIL) consists in learning a policy from a small dataset of action-labeled trajectories and a much larger dataset of action-free trajectories. Some SSIL methods learn an inverse dynamics model (IDM) to predict the action from the current state and the next state. An IDM can act as a policy when paired with a video model (VM-IDM) or as a label generator to perform behavior cloning on action-free data (IDM labeling). In this work, we first show that VM-IDM and IDM labeling learn the same policy in a limit case, which we call the IDM-based policy. We then argue that the previously observed advantage of IDM-based policies over behavior cloning is due to the superior sample efficiency of IDM learning, which we attribute to two causes: (i) the ground-truth IDM tends to be contained in a lower complexity hypothesis class relative to the expert policy, and (ii) the ground-truth IDM is often less stochastic than the expert policy. We argue these claims based on insights from statistical learning theory and novel experiments, including a study of IDM-based policies using recent architectures for unified video-action prediction (UVA). Motivated by these insights, we finally propose an improved version of the existing LAPO algorithm for latent action policy learning. We experiment on the Procgen, Push-T and LIBERO benchmarks.
imitation learninginverse dynamics modelsemi-supervised learninglibero
OpenReview ↗
PDF ↗
Code ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Optimal and Scalable MAPF via Multi-Marginal Optimal Transport and Schrödinger Bridges
Usman Khan, Joseph W Durham
We consider anonymous multi-agent path finding (MAPF) where a set of robots is tasked to travel to a set of targets on a finite, connected graph. We show that MAPF can be cast as a special class of multi-marginal optimal transport (MMOT) problems with an underlying Markovian structure, under which the exponentially large MMOT collapses to a linear program (LP) polynomial in size. Focusing on the anonymous setting, we establish conditions under which the corresponding LP is feasible, totally unimodular, and yields min-cost, integral~$(\{0,1\})$ transports that do not overlap in both space and time. To adapt the approach to large-scale problems, we cast the MAPF-MMOT in a probabilistic framework via Schrödinger bridges. Under standard assumptions, we show that the Schrödinger bridge formulation reduces to an entropic regularization of the corresponding MMOT that admits an iterative Sinkhorn-type solution. The Schrödinger bridge, being a probabilistic framework, provides a shadow (fractional) transport that we use as a template to solve a reduced LP and demonstrate that it results in near-optimal, integral transports at a significant reduction in complexity. Extensive experiments highlight the optimality and scalability of the proposed approaches.
multi-agent path findingoptimal transportmulti-robot coordinationpath planning
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Orchestrating Spatial Semantics via a Zone-Graph Paradigm for Intricate Indoor Scene Generation
Meisheng Zhang, Shizhao Sun, Yang Zhao, Ziyuan Liu, Zhijun Gao, Jiang Bian
Autonomous 3D indoor scene synthesis breaks down in non-convex rooms with tightly coupled spatial constraints. Data-driven generators lack topological priors for long-horizon planning, while iterative agents fragment semantics and become geometrically brittle. We present \textbf{ZoneMaestro}, a unified framework that shifts the paradigm from object-centric synthesis to Zone-Graph Orchestration. By internalizing a novel zone-based logic, ZoneMaestro translates high-level semantic intent into functional zones and topological constraints, enabling robust adaptation to diverse architectural forms. To support this, we construct \textit{Zone-Scene-10K}, a large-scale dataset enriched with explicit Zone-Graph annotations. We further introduce an \textit{Alternating Alignment Strategy} that cycles between reasoning internalization and Zone-Aware Group Relative Policy Optimization (\textit{Z-GRPO}), effectively reconciling the tension between semantic richness and geometric validity without relying on external physics engines. To rigorously evaluate spatial intelligence beyond convex primitives, we formally define the task of \textbf{Intricate Spatial Orchestration} and release SCALE, a stress-test benchmark for irregular indoor scenarios with complex, dense spatial relations. Extensive experiments demonstrate that ZoneMaestro resolves the density-safety dichotomy, significantly outperforming state-of-the-art baselines in both structural coherence and intent adherence.
scene-generationspatial-reasoningindoor-layoutembodied-ai
OXE-AugE: A Large-Scale Robot Augmentation of OXE for Scaling Cross-Embodiment Policy Learning
Guanhua Ji, Harsha Polavaram, Lawrence Yunliang Chen, Sandeep Bajamahal, Zehan Ma, Simeon Adebola, Chenfeng Xu, Ken Goldberg
Large and diverse datasets are needed for training generalist robot policies that can control a variety of robot embodiments--robot arm and gripper combinations--across diverse tasks and environments. As re-collecting demonstrations and retraining for each new embodiment are prohibitively costly, we study whether existing robot data can be augmented to improve transfer and generalization across embodiments. The Open X-Embodiment (OXE) dataset, which aggregates demonstrations from over 60 robot datasets, has been widely used for training generalist policies. However, it is highly imbalanced: the top four robot types account for over 85% of its real data, which risks overfitting to robot--scene combinations. We present AugE-Toolkit, a scalable robot augmentation pipeline, and OXE-AugE, a high-quality open-source dataset that augments OXE with 9 different robot embodiments. OXE-AugE provides over 4.4 million trajectories, more than triple the size of the original OXE. We conduct a systematic study of how scaling robot augmentation impacts cross-embodiment learning. Results suggest that augmenting datasets with diverse arms and grippers improves policy performance not only on the augmented robots, but also on unseen robots and even the original robots under distribution shifts. In physical experiments, fine-tuning generalist policies such as OpenVLA and $\pi_0$ on OXE-AugE improves success rates by 24-45% on unseen robot-gripper combinations across four real-world manipulation tasks. Project website: https://OXE-AugE.github.io/.
cross-embodimentdataset augmentationgeneralist policymanipulation
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 PACT: Self-Evolving Physical Safety Alignment for Diffusion Policies in Embodied Manipulation
Lingxuan Wu, Zijian Zhu, Lizhong Wang, Chengyang Ying, Huayu Chen, Xiao Yang, Fangming Liu, Jun Zhu
Diffusion policies have achieved remarkable success in robotic manipulation, yet they often fail to satisfy strict physical constraints required for safe deployment. Existing approaches impose safety either prematurely during training or reactively via external guardrails at test time, limiting policy expressivity and overall scalability. We propose Physical safety Alignment for Constrained Trajectories (PACT), a self-evolving post-training framework that projects pretrained diffusion policies onto constraint-feasible regions without accessing demonstration data or task rewards. PACT distills constraint gradients into the diffusion model through a reverse-KL objective with dense supervision across timesteps. It incorporates a curriculum that progressively tightens constraints while maintaining theoretically bounded policy shift and monotone improvement, mitigating the safety-performance trade-off from catastrophic forgetting. On simulated and real-world embodied manipulation benchmarks, PACT significantly reduces safety violations by 31.0% on average while improving task success by 30.7%.
diffusion policysafety alignmentembodied manipulationconstraint satisfaction
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 PAWS: Preference Learning with Advantage-Weighted Segments
Aleksandar Taranovic, Onur Celik, Niklas Freymuth, Ge Li, Serge Thilges, Huy Le, Tai Hoang, Rania Rayyes, Gerhard Neumann
Preference-based reinforcement learning (PbRL) learns policies from human trajectory-level comparisons, avoiding explicit reward design and expert demonstrations. Existing methods typically train utility functions on trajectory or segment-level preferences while relying on per-step utility estimates during policy optimization. This training and inference mismatch induces a distribution shift that severely degrades temporal credit assignment and limits policy learning.
We analyze this issue and propose PAWS, a segment-based preference learning method that performs policy updates directly using segment-level advantage functions. By aligning utility training with policy optimization, PAWS preserves trajectory-level preference information and avoids unreliable per-step learning signals. Experiments on simulated robotic manipulation and locomotion tasks demonstrate that PAWS consistently outperforms existing PbRL approaches, highlighting the importance of distribution-consistent preference learning.
preference-based rlrobotic manipulationlocomotioncredit assignment
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 PhaseCoder: Microphone Geometry-Agnostic Spatial Audio Understanding for Multimodal LLMs
Artem Dementyev, Wazeer Zulfikar, Sinan Hersek, Pascal Getreuer, Anurag Kumar, Vivek Kumar
Current multimodal large language models (LLMs) process audio as a mono stream, ignoring the rich spatial information essential for embodied AI. Conversely, existing spatial audio models are constrained to fixed microphone geometries, preventing their deployment across diverse devices. We present **PhaseCoder**, a transformer-only spatial audio encoder that is inherently agnostic to microphone geometry. By taking raw multichannel audio and microphone coordinates as inputs, PhaseCoder performs accurate localization and produces robust spatial embeddings. We demonstrate that the Gemma 3n LLM can be fine-tuned to process and reason over the "Spatial Audio Tokens" produced by our encoder. PhaseCoder achieves state-of-the-art results on microphone-invariant localization benchmarks and, for the first time, enables an LLM to perform complex spatial reasoning and targeted transcription tasks from an arbitrary microphone array.
spatial-audiosound-localizationmultimodal-llmembodied-ai
OpenReview ↗
PDF ↗
Poster Session 8 (Hall A) · Thursday, July 9, 5:00-6:45 PhyScene3D: Physically Consistent 3D Interactive Tabletop Scene Generation
Weixing Chen, Zhuoqian Feng, Yang Liu, Yexin Zhang, Yifan Wen, Yinghong Liao, Weichao Qiu, Guanbin Li, Liang Lin
Generating physically consistent 3D tabletop scenes is a fundamental yet underexplored problem for interactive and generalist robotic learning. The challenge stems from dense object hierarchies and irregular affordances. Here, an interactive scene denotes a physically valid, collision-free environment directly loadable into physics simulators. Existing methods, ranging from decoupled symbolic solvers to end-to-end regression models, often suffer from error propagation or overfitting to noisy supervision containing widespread physical violations. To address these limitations, we introduce PhyScene3D, a framework that reformulates generation as a Human-Mimetic Constructive Process. The proposed Cognitive Topological Reasoning Chain (CTRC) factorizes scene synthesis into a sequential, anchor-conditioned process. It employs a 3D AABB-based placement scheme that imposes a strong structural inductive bias. To address imperfect supervision and physical infeasibility, we introduce Physics-Aware Denoising Alignment (PADA). It integrates a differentiable Signed Distance Field (SDF) with Test-Time Optimization (TTO) to project generated scenes onto a physics-feasible manifold while preserving semantic intent. Experiments demonstrate that PhyScene3D outperforms state-of-the-art approaches in both semantic accuracy and physical validity, achieving a 40\% reduction in scene-wise collision rate relative to the human-annotated training data.
3d scene generationphysics simulationrobot learning environmentstabletop manipulation
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 PhysForge: Generating Physics-Grounded 3D Assets for Interactive Virtual World
Yunhan Yang, Chunshi Wang, Junliang Ye, YANG LI, Zanxin Chen, Zehuan Huang, Yao Mu, Zhuo Chen, Chunchao Guo, Xihui Liu
Synthesizing physics-grounded 3D assets is a critical bottleneck for interactive virtual worlds and embodied AI. Existing methods predominantly focus on static geometry, overlooking the functional properties essential for interaction. We propose that interactive asset generation must be rooted in functional logic and hierarchical physics. To bridge this gap, we introduce PhysForge, a decoupled two-stage framework supported by PhysDB, a large-scale dataset of 150,000 assets with four-tier physical annotations. First, a VLM acts as a physical architect to plan a Hierarchical Physical Blueprint defining material, functional, and kinematic constraints. Second, a physics-grounded diffusion model realizes this blueprint by synthesizing high-fidelity geometry alongside precise kinematic parameters via a novel KineVoxel Injection (KVI) mechanism. Experiments demonstrate that PhysForge produces functionally plausible, simulation-ready assets, providing a robust data engine for interactive 3D content and embodied agents.
3d-asset-generationphysics-groundedarticulationembodied-ai
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 Plan in Sandbox, Navigate in Open Worlds: Learning Physics-Grounded Abstracted Experience for Embodied Navigation
Zhixuan Shen, Jiawei Du, Ziyu Guo, Han Luo, Lilan Peng, Joey Tianyi Zhou, Haonan Luo, Tianrui Li
Vision-Language Models (VLMs) have demonstrated exceptional general reasoning capabilities. However, their performance in embodied navigation remains hindered by a scarcity of aligned open-world vision and robot control data. Despite simulators providing a cost-effective alternative for data collection, the inherent reliance on photorealistic simulations often limits the transferability of learned policies. To this end, we propose ***S**andbox-**A**bstracted **G**rounded **E**xperience* (***SAGE***), a framework that enables agents to learn within a physics-grounded semantic abstraction rather than a photorealistic simulation, mimicking the human capacity for mental simulation where plans are rehearsed in simplified physics abstractions before execution. *SAGE* operates via three synergistic phases: (1) *Genesis*: constructing diverse, physics-constrained semantic environments to bootstrap experience; (2) *Evolution*: distilling experiences through Reinforcement Learning (RL), utilizing a novel asymmetric adaptive clipping mechanism to stabilize updates; (3) *Navigation*: bridging the abstract policy to open-world control. We demonstrate that *SAGE* significantly improves planner-assisted embodied navigation, achieving a 53.21% LLM-Match Success Rate on A-EQA (+9.7% over baseline), while showing encouraging transfer to physical indoor robot deployment. Project page is available at: <https://frankzxshen.github.io/SAGE>.
embodied navigationsim-to-realreinforcement learningvlm planning
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Plug-and-Play Label Map Diffusion for Universal Goal-Oriented Navigation
Zhixuan Shen, Yijie Zeng, Shengxiang Luo, Tianrui Li, Haonan Luo
In embodied vision, Goal-Oriented Navigation (GON) requires robots to locate a specific goal within an unexplored environment. The primary challenge of GON arises from the need to construct a Bird's-Eye-View (BEV) map to understand the environment while simultaneously localizing an unobserved goal. Existing map-based methods typically employ self-centered semantic maps, often facing challenges such as reliance on complete maps or inconsistent semantic association. To this end, we propose Plug-and-Play Label Map Diffusion (PLMD), which defines a novel map completion diffusion model based on Denoising Diffusion Probabilistic Models (DDPM). PLMD generates obstacle and semantic labels for unobserved regions through a diffusion-based completion process, thereby enabling goal localization even in partially observed environments. Moreover, it mitigates inconsistent semantic association by leveraging structural consistency between known and unknown obstacle layouts and integrating obstacle priors into the semantic denoising process. By substituting predicted labels for unobserved regions, robots can accurately localize the specified objects. Extensive experiments demonstrate that PLMD **(I)** effectively expands the region of unknown maps, **(II)** integrates seamlessly into existing navigation strategies that rely on semantic maps, **(III)** achieves state-of-the-art performance on three GON tasks. Code is available at: <https://github.com/FrankZxShen/PLMD>.
goal-oriented navigationsemantic mappingdiffusion modelembodied vision
OpenReview ↗
PDF ↗
Code ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 PolyFlow: Safe and Efficient Polytope-Constrained Flow Matching with Constraint Embedding and Projection-free Update
jianming ma, Qiyue Yang, Yang Zhang, Liyun Yan, Zhanxiang Cao, Yazhou Zhang, Yue Gao
While flow-based generative models have demonstrated strong performance across a wide range of domains, deploying them in safety-critical physical systems remains challenging due to strict constraint requirements. Existing approaches typically enforce safety through post-hoc corrections, which incur substantial computational overhead and may distort the learned distribution. We propose PolyFlow, a polytope-constrained flow matching framework that embeds constraints directly into the model and flow dynamics. PolyFlow introduces a discrete-time flow formulation and a projection-free architecture, which eliminate the numeration error and guarantee strict satisfaction of arbitrary polyhedral constraints, without the need for expensive iterative solvers. Experimental results show that PolyFlow achieves zero constraint violation while maintaining high distributional fidelity across a range of planning and control tasks. Compared to state-of-the-art constrained generation baselines, PolyFlow significantly reduces inference latency and demonstrates a favorable trade-off between safety, efficiency, and generative quality.
constrained flow matchingsafe motion planningsafe robotic controlgenerative planning
OpenReview ↗
PDF ↗
Code ↗
Poster Session 7 (Hall A) · Thursday, July 9, 2:30-4:15 Population-Free Pareto Tracking for Sample-Efficient Multi-Policy MORL
Zeyu Zhao, Yueling Che, Kaichen Liu, Jian Li, Junmei Yao
Multi-objective reinforcement learning (MORL) is a fundamental framework for real-world decision-making problems involving multiple conflicting criteria. Existing multi-policy (MP) methods typically rely on online evolutionary frameworks that maintain large policy populations, leading to high sample complexity and excessive agent–environment interactions. To mitigate these limitations, we present Multi-policy Pareto Front Tracking (MPFT), a framework without a self-evolving population. It leverages an efficient Pareto-tracking mechanism initialized with single-objective extreme policies to trace the Pareto front, and further densifies sparse regions to achieve an accurate approximation of the full Pareto front. MPFT can be seamlessly integrated with advanced offline MORL algorithms, thereby
substantially improving sample efficiency. We evaluate MPFT on six robotic control tasks with up to three objectives and three high-dimensional tasks with more than three objectives. Experimental results show that MPFT outperforms state-ofthe-art baselines in terms of hypervolume and expected utility. It also significantly reduces agent–environment interactions. These results further demonstrate that MPFT serves as a general purpose framework that can seamlessly integrate both online and offline MORL algorithms.
multi-objective-rlpareto-frontsample-efficiencypolicy-optimization
OpenReview ↗
PDF ↗
Poster Session 1 (Hall A) · Tuesday, July 7, 10:30-12:15 Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuning
Andrew Wagenmaker, Perry Dong, Raymond Tsao, Chelsea Finn, Sergey Levine
Standard practice across domains from robotics to language is to first pretrain a policy on a large-scale demonstration dataset, and then finetune this policy, typically with reinforcement learning (RL), in order to improve performance on deployment domains. This finetuning step has proved critical in achieving human or super-human performance, yet while much attention has been given to developing more effective finetuning algorithms, little attention has been given to ensuring the pretrained policy is an effective initialization for RL finetuning. In this work we seek to understand how the pretrained policy affects finetuning performance, and how to pretrain policies in order to ensure they are effective initializations for finetuning. We first show theoretically that standard behavioral cloning (BC) can fail to ensure coverage over the demonstrator's actions, a minimal condition necessary for effective RL finetuning. We then show that if, instead of exactly fitting the observed demonstrations, we train a policy to model the posterior distribution of the demonstrator's behavior given the demonstration dataset, we do obtain a policy that ensures coverage over the demonstrator's actions, enabling more effective finetuning. Furthermore, this policy achieves this while ensuring pretrained performance is no worse than that of the BC policy. We then show this approach is practically implementable with modern generative models and leads to significantly improved RL finetuning performance on both realistic robotic control benchmarks and real-world robotic manipulation tasks, as compared to standard behavioral cloning.
behavioral cloningrl finetuningpretrainingrobotic manipulation
OpenReview ↗
PDF ↗
Poster Session 7 (Hall A) · Thursday, July 9, 2:30-4:15 Predicting What Matters: Robust Generalist Robot Policy Learning via Future Semantic Mask
Yunfan Lou, Xiaowei Chi, Xiaojie Zhang, Zezhong Qian, Chengxuan Li, Rongyu Zhang, Yaoxu Lyu, Guoyu Song, Chuyao Fu, Haoxuan Xu, Pengwei Wang, Shanghang Zhang
World models derived from large-scale video generative pre-training have emerged as a promising paradigm for generalist robot policy learning. However, standard approaches often focus on high-fidelity RGB video prediction, this can result in overfitting to irrelevant factors, such as dynamic backgrounds and illumination changes. These distractions reduce the model's ability to generalize, ultimately leading to unreliable and fragile control policies. To address this, we introduce the Mask World Model (MWM), which leverages video diffusion architectures to predict the evolution of semantic masks instead of pixels. This shift imposes a geometric information bottleneck, forcing the model to capture essential physical dynamics and contact relations while filtering out visual noise. We seamlessly integrate this mask dynamics backbone with a diffusion-based policy head to enable robust end-to-end control. Extensive evaluations demonstrate the superiority of MWM on the LIBERO and RLBench simulation benchmarks, significantly outperforming the state-of-the-art RGB-based world models. Furthermore, real-world experiments and robustness evaluation (via random token pruning) reveal that MWM exhibits superior generalization capabilities and robust resilience to texture information loss.
world modelrobot policy learningsemantic mask predictiongeneralization
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Pretrained Vision-Language-Action Models are Surprisingly Resistant to Forgetting in Continual Learning
Huihan Liu, Changyeon Kim, Bo Liu, Minghuan Liu, Yuke Zhu
Continual learning is a long-standing challenge in robot policy learning, where a policy must acquire new skills over time without catastrophically forgetting previously learned ones. While prior work has extensively studied continual learning in relatively small behavior cloning (BC) policy models trained from scratch, its behavior in modern large-scale pretrained Vision-Language-Action (VLA) models remains underexplored. In this work, we find that pretrained VLAs are remarkably resistant to forgetting compared with smaller policy models trained from scratch. Simple Experience Replay (ER) works surprisingly well on VLAs, sometimes achieving zero forgetting even with a small replay data size. Our analysis reveals that pretraining plays a critical role in downstream continual learning performance: large pretrained models mitigate forgetting with a small replay buffer size while maintaining strong forward learning capabilities. Furthermore, we find that VLAs can retain relevant knowledge from prior tasks despite performance degradation during learning new tasks. This knowledge retention enables rapid recovery of seemingly forgotten skills through finetuning. Together, these insights imply that large-scale pretraining fundamentally changes the dynamics of continual learning, enabling models to continually acquire new skills over time with simple replay.
vision-language-actioncontinual learningcatastrophic forgettingrobot policy
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 PRISM: Learning Realistic Depth via Physics-Grounded Noise Disentanglement with Semantic-Geometric Collaboration
Xiujian Liang, Jiacheng Liu, Mingyang Sun, Qichen He, Anda Cheng, Cewu Lu, Jianhua Sun
Real-world physical sensing exhibits complex, heterogeneous noise patterns that deviate significantly from idealized simulation, posing a fundamental bottleneck for sim-to-real transfer. Existing sensor modelings typically treat depth noise as a monolithic black-box process, overlooking the distinct physical mechanisms that govern different error modalities. In this work, we introduce a physics-grounded paradigm that disentangles monolithic noise into two complementary modalities: sensing invalidation and measurement inaccuracy, enabling a tailored treatment of noise sources based on their physical origins. Building on this insight, we propose PRISM, a tripartite framework that distills 3D Visual Foundation Model features as rich spatial-semantic priors for physics-based reasoning. To address the inherent sparsity and class imbalance of invalidation regions, we develop Hierarchical Positive-Prioritized Supervision, integrating multi-scale positive-weighted objectives with a positive-preserving dynamic hard mining strategy to enforce precise artifact delineation. Extensive benchmarks demonstrate that PRISM achieves state-of-the-art fidelity in noisy depth synthesis. Furthermore, downstream robotic experiments show that PRISM facilitates a 92.1% average success rate in the real world, marking a significant improvement over monolithic baselines.
depth sensingsim-to-realnoise modelingrobotic manipulation
PRISM: Perception Reasoning Interleaved for Sequential Decision Making.
Mohamed Salim AISSI, Clémence Grislain, Clément ROMAC, Laure Soulier, Mohamed CHETOUANI, Olivier Sigaud, Nicolas THOME
Scaling LLM-based embodied agents from text-only environments to complex multimodal settings remains a major challenge. Recent work identifies a perception–reasoning–decision gap in standalone Vision–Language Models (VLMs), which often overlook task-critical information. In this paper, we introduce PRISM, a framework that tightly couples perception (VLM) and decision (LLM) through a dynamic question–answer (DQA) pipeline. Instead of passively accepting the VLM’s description, the LLM critiques it, probes the VLM with goal-oriented questions, and synthesizes a compact image description. This closed-loop interaction yields a sharp, task-driven understanding of the scene. We evaluate PRISM on the ALFWorld and Room-to-Room (R2R) benchmarks. We show that: (1) PRISM significantly outperforms state-of-the-art image-based models, (2) our Interactive goal-oriented perception pipeline yields systematic and substantial gains, and (3) PRISM is fully automatic, eliminating the need for handcrafted questions or answers.
embodied agentsvision-language modelperception-reasoning loopvision-and-language navigation
OpenReview ↗
PDF ↗
Poster Session 8 (Hall A) · Thursday, July 9, 5:00-6:45 ProAct: A Benchmark and Multimodal Framework for Structure-Aware Proactive Response
Xiaomeng Zhu, Fengming ZHU, Weijie Zhou, Ye Tian, Zhenlin Hu, Yufei Huang, Yuchun Guo, Xinyu Wu, Zhengyou Zhang, Fangzhen Lin, Xuantang Xiong
While passive agents merely follow instructions, proactive agents align with higher-level objectives, such as assistance and safety by continuously monitoring the environment to determine when and how to act. However, developing proactive agents is hindered by the lack of specialized resources. To address this, we introduce **ProAct-75**, a benchmark designed to train and evaluate proactive agents across diverse domains, including assistance, maintenance, and safety monitoring. Spanning 75 tasks, our dataset features 91,581 step-level annotations enriched with explicit task graphs. These graphs encode step dependencies and parallel execution possibilities, providing the structural grounding necessary for complex decision-making. Building on this benchmark, we propose **ProAct-Helper**, a reference baseline powered by a Multimodal Large Language Model (MLLM) that grounds decision-making in state detection, and leveraging task graphs to enable entropy-driven heuristic search for action selection, allowing agents to execute parallel threads independently rather than mirroring the human's next step. Extensive experiments demonstrate that ProAct-Helper outperforms strong closed-source models, improving trigger detection mF1 by 6.21\%, saving 0.25 more steps in online one-step decision, and increasing the rate of parallel actions by 15.58\%. Code is available at https://github.com/ZhuXMMM/ProAct.git
proactive-agentsbenchmarkmultimodal-llmtask-monitoring
OpenReview ↗
PDF ↗
Code ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 PSG-Nav: Probabilistic Scene Graph Navigation via Multiverse Decision Making
Rufeng Chen, Yue Chang, Xiaqiang Tang, Hechang Chen, Sihong Xie
Open-vocabulary navigation requires embodied agents to manage significant perception uncertainty stemming from semantic ambiguity and model errors.
However, most existing works settle for local optimal deterministic approaches, depriving complex navigation decision-making over multiple composite possibilities that are critical for globally better solutions.
In this paper, we propose Probabilistic Scene Graph Navigation (PSG-Nav), which constructs a 3D Probabilistic Scene Graph that uses full semantic categorical distributions to account for perception uncertainty.
To efficiently use the local distributions to compose and reason about the optimal navigation landmarks, we propose Multiverse Decision to sample multiple most likely world settings from the joint distribution, and evaluate navigation landmarks based on the compatibility between landmarks and multiverses.
To mitigate false positives due to epistemic uncertainty in open-vocabulary navigation, we introduce the Evidential Experience Calibrator, which enables online lifelong adaptation by cross-validating detections against memories of past successes and failures.
Extensive experiments on widely-used benchmarks MP3D, HM3D, and HSSD demonstrate that PSG-Nav establishes new state-of-the-art results, achieving Success Rates of 66.1%, 44.8%, and 67.9%, respectively.
Code is available at: https://psg-nav.github.io
embodied navigationscene graphuncertainty estimationobject-goal navigation
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 R2R2: Robust Representation for Intensive Experience Reuse via Redundancy Reduction in Self-Predictive Learning
Sanghyeob Song, Donghyeok Lee, Jinsik Kim, Sungroh Yoon
For reinforcement learning in data-scarce domains like real-world robotics, intensive data reuse enhances efficiency but induces overfitting. While prior works focus on critic bias, representation-level instability in Self-Predictive Learning (SPL) under high Update-to-Data (UTD) regimes remains underexplored. To bridge this gap, we propose Robust Representation via Redundancy Reduction (R2R2), a regularization method within SPL. We theoretically identify that standard zero-centering conflicts with SPL's spectral properties and design a non-centered objective accordingly. We verify R2R2 on SPL-native algorithms like TD7. Furthermore, to demonstrate its orthogonality to prior advancements, we extend the state-of-the-art SimbaV2, which originally lacks SPL, by integrating a tailored SPL module, termed SimbaV2-SPL. Experiments across 11 continuous control tasks confirm that R2R2 effectively mitigates overfitting; specifically, at a UTD ratio of 20, it improves TD7 by $\sim$22\% and provides additional gains on top of SimbaV2-SPL, which itself establishes a new state-of-the-art. The code can be found at [this link](https://github.com/songsang7/R2R2).
representation-learningself-predictive-learningsample-efficiencycontinuous-control
OpenReview ↗
PDF ↗
Code ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 RA-VLA: Retrieval-Augmented VLA for Test-Time Adaptation
Sanghwan Jang, Minjin Jeon, Minsoo Kim, Seong Jin Choi, Dongha Kim, Hwanjo Yu
Vision-Language-Action (VLA) models provide a versatile foundation for general robotic manipulation, yet they exhibit significant brittleness when confronted with novel task distributions. While In-Context Imitation Learning (ICIL) offers a training-free alternative, existing frameworks suffer from an adaptation bottleneck that hinders the effective translation of expert context to executable actions. This failure originates from superficial retrieval mechanisms and an inherent behavioral inertia that anchors the policy to its pre-trained priors. To address these limitations, we present RA-VLA, a retrieval-augmented VLA framework that integrates behavior-aligned context retrieval with a grounded execution pipeline. By enforcing faithful adherence to functional cues within a scalable architecture, RA-VLA facilitates seamless task adaptation while preserving inference efficiency. Our empirical evaluations across the LIBERO benchmark and a real-world UR5e environment demonstrate that RA-VLA achieves superior success rates and computational efficiency, establishing a robust framework for training-free robotic adaptation.
vision-language-actionin-context imitation learningtest-time adaptationrobotic manipulation
Ratio-Variance Regularized Policy Optimization
Yu Luo, Shuo Han, Yihan Hu, Lei Lv, Huaping Liu, Fuchun Sun, Jianye HAO, Dong Li
Standard on-policy reinforcement learning relies on heuristic clipping to enforce trust regions, but this mechanism imposes a severe cost by indiscriminately truncating high-return yet high-divergence updates. We demonstrate that explicitly constraining the *policy ratio **variance*** provides a principled local approximation to trust-region constraints, eliminating the need for binary hard clipping. By acting as a distributional ''soft brake'', this approach preserves critical gradient signals from novel discoveries while naturally down-weighting and enabling the reuse of stale, off-policy data. We introduce **R$^2$VPO** (Ratio-Variance Regularized Policy Optimization), which implements this constraint via a primal–dual optimization framework. Extensive evaluations across $7$ LLM scales, spanning both fast and slow reasoning paradigms, and $10$ robotic control tasks demonstrate the generality of the proposed approach. R$^2$VPO achieves substantial performance gains on mathematical reasoning benchmarks, with particularly pronounced improvements on smaller models, while significantly improving sample efficiency. Furthermore, it consistently outperforms PPO baselines in continuous control domains, particularly in sparse-reward and dynamic environments. Together, these findings establish ratio-variance regularization as a principled foundation for stable and data-efficient policy optimization.
policy-optimizationtrust-regionsample-efficiencyrobotic-control
OpenReview ↗
PDF ↗
Poster Session 1 (Hall A) · Tuesday, July 7, 10:30-12:15 RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalization
Songming Liu, Bangguo Li, Kai Ma, Lingxuan Wu, Hengkai Tan, Xiao Ouyang, Hang Su, Jun Zhu
Vision-Language-Action (VLA) models hold promise for generalist robotics but currently struggle with data scarcity, architectural inefficiencies, and the inability to generalize across different hardware platforms. We introduce RDT2, a robotic foundation model built upon a 7B parameter VLM designed to enable zero-shot deployment on novel embodiments for open-vocabulary tasks. To achieve this, we collected one of the largest open-source robotic datasets—over $10,000$ hours of demonstrations in diverse families—using an enhanced, embodiment-agnostic Universal Manipulation Interface (UMI). Our approach employs a novel three-stage training recipe that aligns discrete linguistic knowledge with continuous control via Residual Vector Quantization (RVQ), flow-matching, and distillation for real-time inference. Consequently, RDT2 becomes one of the first models that simultaneously zero-shot generalizes to unseen objects, scenes, instructions, and even robotic platforms. Besides, it outperforms state-of-the-art baselines in dexterous, long-horizon, and dynamic downstream tasks like playing table tennis.
vision-language-actioncross-embodimentfoundation modeldexterous manipulationflow matching
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Reasoning-VLA: An Efficient and Spatial-Guided General Vision-Language-Action Reasoning Model for Autonomous Driving
Dapeng Zhang, Zhenlong Yuan, Zhangquan Chen, Chih-Ting Liao, Yinda Chen, Fei Shen, Qingguo Zhou, Tat-Seng Chua
Vision-Language-Action (VLA) models have recently shown strong decision-making capabilities in autonomous driving. However, existing VLAs often struggle with achieving efficient inference and generalizing to novel autonomous vehicle configurations and driving scenarios. In this paper, we propose Reasoning-VLA, a general and efficient action-generation VLA framework. The proposed model employs a set of learnable action queries, implicitly guided by predefined spatial representations to enhance spatial awareness.
These learnable queries interact with reasoning-enhanced vision–language features to generate continuous action trajectories in parallel. To promote robust generalization, we consolidate eight publicly available autonomous driving datasets into a standardized, Chain-of-Thought reasoning–based, and easy-to-use data format for model training. Leveraging both supervised learning and reinforcement learning fine-tuning, extensive empirical evaluations across multiple benchmarks demonstrate that Reasoning-VLA achieves state-of-the-art performance, strong generalization capability, and the excellent inference speed with parallel decode.
vision-language-actionautonomous drivingspatial reasoningaction generation
OpenReview ↗
PDF ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 Recovering Hidden Reward in Diffusion-Based Policies
Yanbiao Ji, Qiuchang Li, Yuting Hu, Shaokai Wu, Wenyuan XIE, Guodong ZHANG, Qichen He, Deyi Ji, Yue Ding, Hongtao Lu
This paper introduces EnergyFlow, a framework that unifies generative action modeling with inverse reinforcement learning by parameterizing a scalar energy function whose gradient is the denoising field. We establish that under maximum-entropy optimality, the score function learned via denoising score matching recovers the gradient of the expert's soft Q-function, enabling reward extraction without adversarial training. Formally, we prove that constraining the learned field to be conservative reduces hypothesis complexity and tightens out-of-distribution generalization bounds. We further characterize the identifiability of recovered rewards and bound how score estimation errors propagate to action preferences. Empirically, EnergyFlow achieves state-of-the-art imitation performance on various manipulation tasks while providing an effective reward signal for downstream reinforcement learning that outperforms both adversarial IRL methods and likelihood-based alternatives. These results show that the structural constraints required for valid reward extraction simultaneously serve as beneficial inductive biases for policy generalization. The code is available at https://github.com/sotaagi/EnergyFlow.
diffusion policyinverse reinforcement learningmanipulationimitation learning
OpenReview ↗
PDF ↗
Code ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Reflex: Real-Time Vision-Language-Action Control through Streaming Inference
Yuanchun Guo, Bingyan Liu
Flow matching Vision-Language-Action (VLA) models promise precise continuous control, but their iterative denoising nature introduces fundamental incompatibilities with real-time robotics: global timestep injection invalidates KV-caching, forcing a choice between slow $O(N^2)$ re-computation or mathematically incorrect cache reuse.
We present **Reflex**, a framework that enables *real-time streaming inference* for flow matching policies by exploiting the *Timestep-Invariance Property*---that perception encoders are functionally independent of the denoising loop.
Reflex partitions the attention context into static, sliding, and dynamic regions, enabling $O(1)$ incremental cache updates while preserving full-batch-equivalent attention outputs for fixed inputs.
To ensure stability under continuous high-frequency inference, we introduce *AdaRMSNorm*, an adaptive normalization layer that prevents BFloat16 numerical collapse by gating on flow phase.
We further maximize throughput through an *async pipeline* that decouples visual encoding from action generation, combined with *operator fusion* that reduces kernel overhead.
On LIBERO and Kinetix benchmarks, Reflex achieves a 2.58$\times$ inference speedup and 50Hz stable streaming, reducing reaction latency by up to 54\% and enabling efficient deployment without performance degradation.
real-time controlstreaming inferenceflow matchingvision-language-action
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 ReLAM: Learning Anticipation Model for Rewarding Visual Robotic Manipulation
Nan Tang, Jing-Cheng Pang, Guanlin Li, Chao Qian, Yang Yu
Reward design remains a critical bottleneck in visual reinforcement learning (RL) for robotic manipulation. In simulated environments, rewards are conventionally designed based on the distance to a target position. However, such precise positional information is often unavailable in real-world visual settings due to sensory and perceptual limitations. In this study, we propose a method that implicitly infers spatial distances through keypoints extracted from images. Building on this, we introduce Reward Learning with Anticipation Model (ReLAM), a novel framework that automatically generates dense, structured rewards from action-free video demonstrations. ReLAM first learns an anticipation model that serves as a planner and proposes intermediate keypoint-based subgoals on the optimal path to the final goal, creating a structured learning curriculum directly aligned with the task's geometric objectives. Based on the anticipated subgoals, a continuous reward signal is provided to train a low-level, goal-conditioned policy under the hierarchical reinforcement learning (HRL) framework with provable sub-optimality bound. Extensive experiments on complex, long-horizon manipulation tasks show that ReLAM significantly accelerates learning and achieves superior performance compared to SOTA methods.
visual manipulationreward learninghierarchical reinforcement learningkeypoints
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Reparameterization Flow Policy Optimization
Hai Zhong, Zhuoran Li, Xun Wang, Longbo Huang
Reparameterization Policy Gradient (RPG) has emerged as a powerful paradigm for model-based reinforcement learning, enabling high sample efficiency by backpropagating gradients through differentiable dynamics. However, prior RPG approaches have been predominantly restricted to Gaussian policies, limiting their performance and failing to leverage recent advances in generative models. In this work, we identify that flow policies, which generate actions via differentiable ODE integration, naturally align with the RPG framework, a connection not established in prior work. Yet, naively realizing this synergy is non-trivial: it often suffers from training instability and a lack of exploration. We propose Reparameterization Flow Policy Optimization (RFO). RFO computes policy gradients by backpropagating jointly through the flow generation process and system dynamics, unlocking high sample efficiency without requiring intractable log-likelihood calculations. RFO includes two tailored regularization terms for stability and exploration. We also propose a variant of RFO with action chunking. Extensive experiments on diverse locomotion and manipulation tasks—involving both rigid and soft bodies with state or visual inputs—demonstrate the effectiveness of RFO. Notably, on a challenging locomotion task controlling a soft-body quadruped, RFO achieves almost $2\times$ the reward of the state-of-the-art baseline.
model-based RLflow policylocomotionquadrupedmanipulation
OpenReview ↗
PDF ↗
Poster Session 1 (Hall A) · Tuesday, July 7, 10:30-12:15 Representation Learning for Equivariant Inference with Guarantees
Daniel Ordonez-Apraez, Vladimir R Kostic, Alek Fröhlich, Vivien Brandt, Karim Lounici, Massimiliano Pontil
In many real-world applications of regression, conditional probability estimation, and uncertainty quantification, exploiting symmetries rooted in physics or geometry can dramatically improve generalization and sample efficiency. While geometric deep learning has made empirical advances by incorporating symmetry and geometry priors, less attention has been given to statistical learning guarantees. In this paper, we introduce an equivariant representation learning framework that simultaneously addresses regression, conditional probability estimation, and uncertainty quantification while providing first-of-its-kind non-asymptotic statistical learning guarantees. Grounded in operator and group representation theory, our framework approximates the spectral decomposition of the conditional expectation operator, building representations that are both equivariant and disentangled along independent symmetry quotient groups. Empirical evaluations on synthetic datasets and real-world robotics applications confirm the potential of our approach, matching or outperforming existing equivariant baselines in regression while providing well-calibrated uncertainty estimates.
equivariancerepresentation-learningstatistical-guaranteesgeometric-deep-learning
OpenReview ↗
PDF ↗
Code ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Rethinking Instruction Drift as a Sampling Error: SNR-Aware Power Distributions for Long-Horizon Robotic Planning
Kewei Chen, Yayu Long, Mingsheng Shang
Despite rapid progress in Vision-Language-Action (VLA) models for robotic control, instruction drift remains a persistent failure mode in long-horizon tasks. This paper reconceptualizes this phenomenon, positing that instruction drift is fundamentally a systematic sampling error: local greedy sampling is prone to collapsing into “Negative Pivotal Windows”—irreversible local optima with high local probability that sever global success pathways. To address this, we propose Context-Aware Power Sampling (CAPS), a training-free inference-time computation framework. CAPS leverages power distributions to sharpen global trajectory probabilities, enabling lookahead search over the model's conditional generative trajectory distribution. Furthermore, we introduce a metacognitive control mechanism based on Signal-to-Noise Ratio (SNR). This mechanism triggers adaptive MCMC search solely when drift risk is detected, enabling a dynamic transition from “intuitive fast thinking” to “rational slow search.” Experiments on RoboTwin, Simpler-WindowX, and Libero-long benchmarks show that CAPS achieves substantial improvements over strong baselines, including OpenVLA and TACO, without parameter updates. These results support the effectiveness of adaptive inference-time computation for improving long-horizon robustness in embodied control.
vision-language-actionlong-horizon planninginference-time searchembodied control
Rethinking Video Generation Model for the Embodied World
Yufan Deng, Zilin Pan, Hongyu Zhang, Xiaojie Li, Ruoqing Hu, Yufei Ding, Yiming Zou, Yan Zeng, Daquan Zhou
While video generation holds promise for embodied intelligence, current video models struggle with physical realism, and progress is hindered by the lack of standardized benchmarks. To address this gap, we introduce a comprehensive robotics benchmark, RBench, designed to evaluate robot-oriented video generation across five task domains and four distinct embodiments. By assessing task correctness and visual fidelity through reproducible metrics, our evaluation of 25 models reveals significant deficiencies in generating physically realistic robot behaviors. Furthermore, the benchmark achieves a 0.96 Spearman correlation with human judgment, validating its effectiveness. While RBench provides the necessary lens to identify these deficiencies, achieving physical realism requires moving beyond evaluation to address the critical shortage of high-quality training data. Driven by these insights, we introduce a refined four-stage data pipeline, resulting in RoVid-X, the largest open-source robotic dataset for video generation with 4 million annotated video clips, covering thousands of tasks and enriched with physical property annotations. Extensive experiments demonstrate that finetuning on RoVid-X yields consistent performance gains. Collectively, this synergistic ecosystem of evaluation and data establishes a robust foundation for rigorous assessment and scalable training of video models, accelerating the evolution of embodied AI toward physical intelligence. The code and video demos are available in the supplementary materials.
video generationrobot benchmarkembodied AIphysical realism
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 Reward Learning through Ranking Mean Squared Error
Chaitanya Kharyal, Calarina Muslimani, Matthew E. Taylor
Reward design remains a significant bottleneck in applying reinforcement learning (RL) to real-world problems.
A popular alternative is reward learning, where reward functions are inferred from human feedback rather than manually specified.
Recent work has proposed learning reward functions from human ratings rather than traditional binary preferences, enabling richer and potentially less cognitively demanding supervision.
Building on this paradigm, we introduce a new rating-based RL method, Ranked Return Regression for RL (R4).
At its core, R4 uses a novel ranking mean squared error loss that learns from a dataset of trajectory–rating pairs, treating the human-provided discrete ratings (e.g., "bad," "neutral," "good") as ordinal targets.
Unlike prior rating-based approaches, R4 offers formal guarantees: its solution set is provably minimal and complete under mild assumptions. Empirically, using both human-provided and simulated ratings, we demonstrate that R4 consistently matches or outperforms existing rating and preference-based RL methods on robotic benchmarks from OpenAI Gym and the DeepMind Control Suite.
reward-learninghuman-feedbackrating-based-rlpreference-learning
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 8 (Hall A) · Thursday, July 9, 5:00-6:45 RoboFlow4D: A Lightweight Flow World Model Toward Real-Time Flow-Guided Robotic Manipulation
Sixu Lin, Junliang Chen, Huaiyuan Xu, Zhuohao Li, Guangming Wang, Yixiong Jing, Sheng Xu, Runyi Zhao, Brian Sheil, Lap-Pui Chau, Guiliang Liu
Planning and acting in 3D environments is a fundamental capability for robotic manipulation in the real world.
Although prior work has explored predictive flow planners to guide 3D manipulation, existing approaches often rely on modular pipelines stacking multiple submodels, resulting in high computational overhead and limited real-time performance. To address these challenges, we introduce RoboFlow4D, a lightweight flow world model that unifies perception and planning by estimating temporal motion in physical 3D space. As an end-to-end framework, RoboFlow4D directly predicts multi-frame 3D flows from visual observations and textual instructions, providing explicit flow-based planning to guide action generation. This design allows seamless integration with general action policies, forming an efficient observation–planning–execution closed loop. Through slow–fast collaboration between flow prediction and action control, RoboFlow4D enables real-time and resource-efficient manipulation. Extensive experiments in both simulation and real-world settings demonstrate that RoboFlow4D consistently improves manipulation success rates and computational efficiency, advancing flow-guided planning for embodied intelligence. Our project page is available at RoboFlow4D.
flow world model3D manipulationreal-time planningembodied intelligence
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies
Yinpei Dai, Hongze Fu, Jayjun Lee, Yuejiang Liu, Haoran Zhang, Jianing Yang, Chelsea Finn, Nima Fazeli, Joyce Chai
Memory is critical for long-horizon and history-dependent robotic manipulation. Such tasks often involve counting repeated actions or manipulating objects that become temporarily occluded. Recent vision-language-action (VLA) models have begun to incorporate memory mechanisms; however, their evaluations remain confined to narrow, non-standardized settings. This limits systematic understanding, comparison, and progress measurement. To address these challenges, we introduce **RoboMME**: a large-scale standardized benchmark for evaluating and advancing VLA models in long-horizon, history-dependent scenarios. Our benchmark comprises 16 manipulation tasks constructed under a carefully designed taxonomy that evaluates *temporal*, *spatial*, *object*, and *procedural* memory. We further develop a suite of 14 memory-augmented VLA variants built on the $\pi_{0.5}$ backbone to systematically explore different memory representations across multiple integration strategies. We show that the effectiveness of memory representations is highly task-dependent, with each design offering distinct advantages and limitations across different tasks. Videos and code can be found at https://robomme.github.io
memory-augmented policyvision-language-actionbenchmarklong-horizon manipulation
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 RoboOmni: Actions Are Just Another Modality for Vision-Language Models
Dong Wang, Zilong Chen, Jirong Liu, Ziqing Qiao, Xin Xiao, Bingyi Kang, Hongtao Wu, Xiao Ma, Tao Kong, Huaping Liu
Integrating Vision-Language Models (VLMs) into robotics has facilitated the development of generalizable Vision-Language Action (VLA) policies.
However, unified discrete frameworks lag behind decoupled continuous designs due to limitations in action chunking and temporal modeling.
To address this, we introduce **RoboOmni**, a unified multi-modal next-token prediction framework.
Challenging the assumption that continuous modeling is essential for high-performance manipulation, **RoboOmni** demonstrates that *actions are just another modality* capable of being effectively modeled discretely.
At the core of our method is Multi-Token Action Prediction (MTAP), which integrates action chunking directly into the discrete tokenizer. This design resolves temporal modeling bottlenecks and significantly reduces distribution shift between training and inference. By preserving the native VLM training and inference pipeline, **RoboOmni** naturally benefits from large-scale multimodal co-training and modern decoding optimizations.
Extensive evaluations on the CALVIN, SimplerEnv, and real-world platforms confirm that **RoboOmni** establishes new state-of-the-art performance, significantly outperforming diffusion-based baselines such as $\pi_0$.
Notably, combining our proposed MTAP with the FAST tokenizer achieves a 94.4\% average success rate on CALVIN,
while the Bin tokenizer implementation attains a 27$\times$ inference speedup compared to OpenVLA.
vision-language-actionaction tokenizationmanipulationmultimodal
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation
Tianxing Chen, Zanxin Chen, Baijun Chen, Zijian Cai, Yibin Liu, Zixuan Li, Qiwei Liang, Xianliang Lin, Yiheng Ge, Zhenyu Gu, Weiliang Deng, Yubin Guo, Tian Nian, Xuanbing Xie, Qiangyu Chen, Kailun Su, Tianling Xu, Guodong Liu, Mengkang Hu, Huan-ang Gao, Kaixuan Wang, Zhixuan Liang, Yusen Qin, Xiaokang Yang, Ping Luo, Yao Mu
Simulation-based data synthesis has emerged as a powerful paradigm for enhancing real-world robotic manipulation. However, existing synthetic datasets remain insufficient for robust bimanual manipulation due to two key challenges: (1) the lack of an autonomous self-correcting mechanism to resolve execution failures in complex coordination tasks, and (2) the scarcity of diverse visual and spatial variations required to bridge the sim-to-real gap. To this end, we present RoboTwin 2.0, a scalable simulation framework that enables closed-loop, automated, large-scale generation of diverse and realistic data, along with unified evaluation protocols for dual-arm manipulation. Built upon RoboTwin-OD, a foundational library of 731 instances across 147 categories with rich semantic annotations, our framework integrates Multimodal Large Language Models (MLLMs) with simulation-in-the-loop verification. This integration forms an automated feedback mechanism that significantly boosts the success rate of expert task program generation. To enhance robust sim-to-real transfer, RoboTwin 2.0 incorporates structured domain randomization along five axes: clutter, lighting, background, tabletop height and language instructions, thereby maximizing data diversity. We instantiate this framework across 50 dual-arm tasks spanning five robot embodiments. Empirical evaluations demonstrate that Vision-Language-Action (VLA) models pre-trained on our synthetic data achieve a 3.6x improvement in few-shot real-world transfer (over a 10-demo baseline) and a 2.2x gain in zero-shot generalization. We release the data generator, benchmark, pre-collected dataset, and code to support scalable research in robust bimanual manipulation.
bimanual manipulationdomain randomizationsynthetic data generationbenchmark
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 SAD-Flower: Flow Matching for Safe, Admissible, and Dynamically Consistent Planning
Tzu-Yuan Huang, Armin Lederer, Dai-Jie Wu, Xiaobing Dai, Sihua Zhang, Hsiu-Chin Lin, Shao-Hua Sun, Stefan Georg Sosnowski, Sandra Hirche
Flow matching (FM) has shown promising results in data-driven planning. However, it inherently lacks formal guarantees for ensuring state and action constraints, whose satisfaction is a fundamental and crucial requirement for the safety and admissibility of planned trajectories on various systems. Moreover, existing FM planners do not ensure the dynamical consistency, which potentially renders trajectories inexecutable. We address these shortcomings by proposing SAD-Flower, a novel framework for generating $\textbf{S}$afe, $\textbf{A}$dmissible, and $\textbf{D}$ynamically consistent trajectories. Our approach relies on an augmentation of the flow with a virtual control input. Thereby, principled guidance can be derived using techniques from nonlinear control theory, providing formal guarantees for state constraints, action constraints, and dynamic consistency. Crucially, SAD-Flower operates without retraining, enabling test-time satisfaction of unseen constraints. Through extensive experiments across several tasks, we demonstrate that SAD-Flower outperforms various generative-model-based baselines in ensuring constraint satisfaction. Video and demos can be found at [sadflowerplanning.github.io](https://sadflowerplanning.github.io/project_web/).
flow matchingsafe planningconstraint satisfactionrobotic trajectories
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 SafeDec: Constrained Decoding for Safe Autoregressive Generalist Robot Navigation Policies
Parv kapoor, Akila Ganlath, Michael Clifford, Changliu Liu, Sebastian Scherer, Eunsuk Kang
Recent advances in end-to-end, multi-task robot policies based on transformer models have demonstrated impressive generalization to real-world embodied navigation tasks. Trained on vast datasets of simulated and real-world trajectories, these policies map multimodal observations directly to action sequences for physical execution. Despite promising real-world capabilities, these models are still data-driven and, therefore, lack explicit notions of behavioral correctness. We address this gap by introducing **SafeDec**, a constrained decoding framework for autoregressive, transformer-based robot navigation policies that enforces safety specifications expressed as Signal Temporal Logic (STL) formulas. Our method ensures that generated actions provably satisfy STL specifications under assumed dynamics at runtime without retraining while remaining agnostic of the underlying policy.
We evaluate **SafeDec** on tasks from the CHORES benchmark for state-of-the-art embodied navigation policies across hundreds of procedurally generated environments and show that our decoding-time interventions are useful not only for filtering unsafe actions but also for conditional action generation. Videos are available at constrained-robot-fms.github.io.
constrained decodingsignal temporal logicrobot navigationsafety specification
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 SafeLab: An Interactive High-Fidelity Benchmark for Embodied Safety in Scientific Robotics
Fengshuo Bai, Yufeng Li, Ruihai Wu, Peishuo Wang, Yuhan Wang, Bernie Hao Zhu, Yuanfei Wang, Tawei Chou, Jing Gao, Runchuan Zhu, Ying Wen, Yaodong Yang, Yuanpei Chen
Scientific embodied agents could automate laboratory workflows, but laboratory success is trajectory-level: an agent must remain safe throughout execution, not merely reach a final goal. In benchtop settings a small pose, force, or tilt error can cause irreversible spillage or equipment damage, yet most robot benchmarks evaluate reversible, high-tolerance manipulation and imitation-trained policies receive no recovery signal for execution drift. We introduce SafeLab, a generative simulation benchmark that couples a verified generative engine, an automated expert for teleoperation-free demonstrations, and a safety-aware RL interface, with 63 calibrated laboratory assets, 64 tasks across 9 manipulation categories, and 6,400 expert trajectories. Across state-of-the-art policies, it reveals unsafe-but-successful trajectories—large gaps between task success and Safe Success Rate—most acutely in liquid handling, force-limited actuation, and bimanual glassware rearrangement. Bounded residual RL then learns execution-level corrections that raise the simulated Safe Success Rate by up to 43.0 percentage points without retraining the base policy. Finally, 50 open-loop physical replays show 86% agreement between simulated and observed safety outcomes, supporting SafeLab as a scalable platform for screening and training safer laboratory agents.
embodied safetylaboratory roboticsbenchmarkreinforcement learning
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Sample from What You See: Visuomotor Policy Learning via Diffusion Bridge with Observation-Embedded Stochastic Differential Equation
Zhaoyang Liu, Mokai Pan, Zhongyi Wang, Kaizhen Zhu, Haotao Lu, Haipeng Zhang, Jingya Wang, Ye Shi
Imitation learning with diffusion models has advanced robotic control by capturing the multi-modal action distributions. However, existing methods typically treat observations only as high-level conditions to the denoising network, rather than integrating them into the stochastic dynamics of the diffusion process itself. As a result, the sampling is forced to begin from random noise, weakening the coupling between perception and control and often yielding suboptimal performance. We propose BridgePolicy, a generative visuomotor policy that directly integrates observations into the stochastic dynamics via a diffusion-bridge formulation. By constructing an observation-informed trajectory, BridgePolicy enables sampling to start from a rich and informative prior rather than random noise, substantially improving precision and reliability in control. A key difficulty is that diffusion bridge normally connects distributions of matched dimensionality, while robotic observations are heterogeneous and not naturally aligned with actions. To overcome this, we introduce a semantic aligner to unify the visual and state inputs and align the observations with action representations, making diffusion bridge applicable to heterogeneous robot data. Extensive experiments across 52 simulation tasks on three benchmarks and 5 real-world tasks demonstrate that BridgePolicy consistently outperforms state-of-the-art generative policies. Our code is available at https://github.com/jianghcsr/BridgePolicy.
visuomotor policydiffusion bridgeimitation learningrobot manipulation
OpenReview ↗
PDF ↗
Code ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Sample-Efficient Diffusion-based Reinforcement Learning with Critic Guidance
Shutong Ding, Zejia Zhong, Zhongyi Wang, Ke Hu, Bikang Pan, Jingya Wang, Ye Shi
Recent advances in reinforcement learning (RL) have achieved great successes by leveraging the multimodality and exploration capability of diffusion policies. Among these approaches, one representative branch focuses on the sampling-based policy optimization. This design enables better exploration capability of the diffusion model, particularly at the beginning of training, but suffer from low exploitation in Q-value information, resulting in a slow policy convergence. Another branch pays attention to gradient-based policy optimization, which sufficiently exploits the gradient of the Q function yet tends to collapse into a unimodal policy with low diversity. To address this issue, we propose CGPO, \textbf{C}ritic-\textbf{G}uided diffusion \textbf{P}olicy \textbf{O}ptimization, which effectively balances exploration and exploitation with the training-free guidance technique integrated into the denoising process of diffusion policy. Concretely, CGPO steers action generation toward high-value regions defined by the critic network and uses the guided actions as regression objectives. In this manner, CGPO reduces the time required to obtain high-quality actions and improves final performance with better balance between the exploration-exploitation tradeoff. We validate the effectiveness of CGPO on 5 MuJoCo locomotion tasks, and CGPO achieves state-of-the-art performance compared with existing diffusion-based RL methods. Notably, CGPO is the first success to incorporate diffusion policy into real-world RL, with its superior performance on Franka robot arm grasping tasks. Our official page is released at \url{https://dingsht.tech/cgpo-webpage}.
diffusion policyrobotic graspinglocomotioncritic-guided RL
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 SC$^{2}$-WM: A Self-Correcting World Model with Closed-Loop Feedback for Vision-and-Language Navigation in Continuous Environments
Xuan Yao, Yuze Zhu, Junyu Gao, Zongmeng Wang, Changsheng Xu
Vision-and-Language Navigation in Continuous Environments (VLN-CE) requires agents to make fine-grained navigation decisions under partial observability.
However, most existing methods rely on open-loop execution, lacking mechanisms to detect and correct internal state drift during inference.
We propose SC$^{2}$-WM, a self-correcting world model framework that introduces internal feedback for closed-loop decision making in VLN-CE.
Our method derives feedback from world-model foresight to perform state-level plan refinement before action execution.
To handle challenging scenarios, we further introduce conditional world-aware adaptation, which enables model-level correction by selectively updating the world model at test time when feedback indicates model capacity insufficiency.
Experiments on standard VLN-CE benchmarks demonstrate improved navigation robustness and generalization. Our code is available at https://github.com/sunrise-ikun/SC2_WM.
vision-and-language navigationworld modelclosed-loop controlembodied AI
OpenReview ↗
PDF ↗
Code ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 Scalable and General Whole-Body Control for Cross-Humanoid Locomotion
Yufei Xue, Yunfeng Lin, Wentao Dong, Yang Tang, Jingbo Wang, Jiangmiao Pang, Ming Zhou, Minghuan Liu, Weinan Zhang
Learning-based whole-body controllers have become a key driver for humanoid robots, yet most existing approaches require robot-specific training.
In this paper, we study the problem of cross-embodiment humanoid control and show that a single policy can robustly generalize across a wide range of humanoid robot designs with one-time training.
We introduce XHugWBC, a novel cross-embodiment training framework that enables generalist humanoid control through:
(1) physics-consistent morphological randomization,
(2) semantically aligned observation and action spaces across diverse humanoid robots, and
(3) effective policy architectures modeling morphological and dynamical properties.
XHugWBC is not tied to any specific robot. Instead, it internalizes a broad distribution of morphological and dynamical characteristics during training. By learning motion priors from diverse randomized embodiments, the policy acquires a strong structural bias that supports zero-shot transfer to previously unseen robots.
Experiments on twelve simulated humanoids and seven real-world robots demonstrate the strong generalization and robustness of the resulting universal controller.
whole-body controlcross-embodimentlocomotionsim-to-real
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 SCALE: Self-uncertainty Conditioned Adaptive Looking and Execution for Vision-Language-Action Models
Hyeonbeom Choi, Daechul Ahn, Youhan Lee, Taewook Kang, Seongwon Cho, Jonghyun Choi
Vision-Language-Action (VLA) models have emerged as a promising paradigm for general-purpose robotic control, with test-time scaling (TTS) gaining attention to enhance robustness beyond training. However, existing TTS methods for VLAs require additional training, verifiers, and multiple forward passes, making them impractical for deployment. Moreover, they intervene only at action decoding while keeping visual representations fixed—insufficient under perceptual ambiguity, where reconsidering how to perceive is as important as deciding what to do. To address these limitations, we propose SCALE, a simple inference strategy that jointly modulates visual perception and action based on 'self-uncertainty', inspired by uncertainty-driven exploration in Active Inference theory—requiring no additional training, no verifier, and only a single forward pass. SCALE broadens exploration in both perception and action under high uncertainty, while focusing on exploitation when confident—enabling adaptive execution across varying conditions. Experiments on simulated and real-world benchmarks demonstrate that SCALE improves state-of-the-art VLAs and outperforms existing TTS methods while maintaining single-pass efficiency. Our code is publicly available at https://github.com/snumprlab/scale.
vision-language-actiontest-time scalinguncertainty estimationactive perception
OpenReview ↗
PDF ↗
Code ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 ScaleMoE: Mixture-of-Experts for Scalable Continuous Control in Actor-Critic Reinforcement Learning
Yi Ma, Chenjun Xiao, Hongyao Tang, Yaodong Yang, Jinyi Liu, Jing Liang, Jiye Liang
Scaling network remains a bottleneck in deep reinforcement learning (RL): simply enlarging actor–critic networks destabilizes training and soon saturates performance. Although recent monolithic architectures such as SimBa and BRC have shown that carefully designed inductive biases can enable positive scaling up to a certain size, their improvements plateau soon as model parameters grow further. This work introduces ScaleMoE, a scalable RL architecture that integrates Mixture-of-Experts (MoE) modules into both the actor and critic of modern continuous control algorithms. Two complementary gating schemes are studied: output-level aggregation of per-expert policies and Q-functions, and feature-level fusion of expert representations before a shared head. We instantiate ScaleMoE on two representative monolithic RL baselines: the single-task method SimBa and the multi-task method BRC. Experiments across the DeepMind Control Suite, MetaWorld, and HumanoidBench show that progressively increasing the number of experts (up to 64) yields substantial improvements in returns, significantly outperforming monolithic networks of comparable or even greater parameter counts. Results demonstrate that ScaleMoE provides an efficient and effective scaling axis for deep RL in continuous control.
mixture-of-expertsactor-criticcontinuous-controlscalability
OpenReview ↗
PDF ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 Scaling by Diversified Experience for Vision-Language-Action Models
Leiyu Wang, Zhaofengnian Wang, Xueqi Li, Luoyi Fan, Cewu Lu, Nanyang Ye
Vision-Language-Action models face significant challenges in real-world deployment due to the entanglement of high-level reasoning with low-level control, and the instability of policy optimization. In this paper, we introduce SyVLA, a robust VLA model trained with diversified experiences. We propose an Intention Decoupling algorithm to isolate control-relevant features from reasoning contexts and a similar-sample guided RL pipeline to stabilize policy updates and mitigate distribution shift. Extensive experiments on real-world robotic tasks and multi-modal benchmarks demonstrate that SyVLA achieves superior task success rates and stronger out-of-distribution generalization compared to existing methods, while effectively preserving core vision-language capabilities.
vision-language-actionreinforcement learninggeneralizationreal-world robots
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Scaling Real-World Robot Policy Evaluation via Discrete Diffusion World Model
Yaxuan Li, Junjie Wen, Zhongyi Zhou, Yefei Chen, Chaomin Shen, Yaxin Peng, Yichen Zhu
Evaluating generalist robot manipulation policies is costly and difficult to scale in the real world. While emerging world models (e.g., WorldEval, Ctrl-World) offer a promising alternative, the reliability of such evaluation remains a critical bottleneck.
Specifically, their visual predictions can undermine policy assessment by "self-correcting" failures into false positives or yielding artifacts under out-of-distribution controls.
Even with failure-enriched data, current architectures struggle to capture action-causal dynamics, as they typically treat actions as passive conditions rather than causal drivers.
To address this, we propose dWorldEval, an action-centric discrete-diffusion world model that maps visual observations, language instructions, and action chunks into a shared unified token space and denoises them with a single self-attention backbone where actions function as first-class tokens.
To realize reliable policy-world interaction, dWorldEval introduces a sparse keyframe memory that anchors global scene state while preserving fine-grained multi-view interaction cues, and leverages Progress-as-text to jointly generate future observations and success indicators.
Extensive experiments on LIBERO, RoboTwin, and real-robot tasks demonstrate that dWorldEval significantly outperforms video diffusion baselines in action controllability, stabilizes long-horizon multi-view rollouts, enabling accurate policy ranking via automatic success estimation.
policy evaluationworld modelrobot manipulationdiffusion model
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes
Nicholas Pfaff, Thomas Cohn, Sergey Zakharov, Rick Cory, Russ Tedrake
Simulation has become a key tool for training and evaluating home robots at scale, yet existing environments fail to capture the diversity and physical complexity of real indoor spaces. Current scene synthesis methods produce sparsely furnished rooms that lack the dense clutter, articulated furniture, and physical properties essential for robotic manipulation. We introduce SceneSmith, a hierarchical agentic framework that generates simulation-ready indoor environments from natural language prompts. SceneSmith constructs scenes through successive stages—from architectural layout to furniture placement to small object population—each implemented as an interaction among VLM agents: designer, critic, and orchestrator. The framework tightly integrates asset generation through text-to-3D synthesis for static objects, dataset retrieval for articulated objects, and physical property estimation. SceneSmith generates 3-6x more objects than prior methods, with $<$2\% inter-object collisions and 96\% of objects remaining stable under physics simulation. In a user study with 205 participants, it achieves 92\% average realism and 91\% average prompt faithfulness win rates against baselines. We further demonstrate that these environments can be used in an end-to-end pipeline for automatic robot policy evaluation.
scene generationrobot simulationpolicy evaluationindoor environments
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 SCOPE: Evolving Symbolic World for Planning in Open-Ended Environments
Yundaichuan Zhan, Minghe Gao, Zhongqi Yue, Wendong Bu, Wenqiao Zhang, Guoming Wang, Jisheng Dang, Juncheng Li, Siliang Tang, Yueting Zhuang
Recent works have explored integrating Vision-Language Models (VLMs) with classical planners that rely on symbolic representations of planning problem to generate long-horizon plans for complex embodied tasks. However, in open-ended environments, these symbolic representations obtained from perception are often incomplete, leading to suboptimal performance. To address this, we introduce SCOPE, a self-adaptive symbolic planning framework that supports refining action plans and evolving the symbolic world—the symbolic representations of open-ended environments. SCOPE comprises two synergistic modules: a Symbolic Execution Simulator (SESim) that conducts symbolic validation and real execution of action plans, leveraging the feedback to refine the plans and evolve the symbolic world; and a Self-Adaptive Symbolic Memory (SASMem) that further distills feedback into evolving symbolic knowledge to enhance long-horizon planning and modeling of the symbolic world. Experiments in open-ended environments show that SCOPE significantly improves the completeness of the symbolic world, the success rate of plans under environment perturbations, and cross-task grounding and adaptability across diverse embodied scenarios.
neuro-symbolic planningembodied AIvision-language modelslong-horizon planning
OpenReview ↗
PDF ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 See What Matters: Differentiable Grid Sample Pruning for Generalizable Vision-Language-Action Model
Yixu Feng, Zinan Zhao, Yanxiang Ma, Chenghao Xia, Chengbin Du, Yunke Wang, Chang Xu
Vision-Language-Action (VLA) models have shown remarkable promise in robotics manipulation, yet their high computational cost hinders real-time deployment. Existing token pruning methods suffer from a fundamental trade-off: aggressive compression using pruning inevitably discards critical geometric details like contact points, leading to severe performance degradation. This forces a compromise, limiting the achievable compression rate and thus the potential speedup. We argue that breaking this trade-off requires rethinking compression as a geometry-aware, continuous token resampling in the vision encoder.
To this end, we propose the *Differentiable Grid Sampler (GridS)*, a plug-and-play module that performs task-aware, continuous resampling of visual tokens in VLA. By adaptively predicting a minimal set of salient coordinates and extracting features via differentiable interpolation, GridS preserves essential spatial information while achieving drastic compression (with fewer than 10\% original visual tokens). Experiments on both LIBERO benchmark and a real robotic platform demonstrate that validating the lowest feasible visual token count reported to date,
GridS achieves a 76\% reduction in FLOPs with no degradation in the success rate. The code is available at https://github.com/Fediory/Grid-Sampler.
token pruningvision-language-actionmanipulationefficient inference
OpenReview ↗
PDF ↗
Code ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Seeing Realism from Simulation: Efficient Video Transfer for Vision-Language-Action Data Augmentation
Chenyu Hui, Xiaodi Huang, Siyu Xu, Yunke Wang, Shan You, Fei Wang, Tao Huang, Chang Xu
Vision-language-action (VLA) models typically rely on large-scale real-world videos, whereas simulated data, despite being inexpensive and highly parallelizable to collect, often suffers from a substantial visual domain gap and limited environmental diversity, resulting in weak real-world generalization. We present an efficient video augmentation framework that converts simulated VLA videos into realistic training videos while preserving task semantics and action trajectories. Our pipeline extracts structured conditions from simulation via video semantic segmentation and video captioning, rewrites captions to diversify environments, and uses a conditional video transfer model to synthesize realistic videos. To make augmentation practical at scale, we introduce a diffusion feature-reuse mechanism that reuses video tokens across adjacent timesteps to accelerate generation, and a coreset sampling strategy that identifies a compact, non-redundant subset for augmentation under limited computation.Extensive experiments on Robotwin 2.0, LIBERO, LIBERO-Plus, and a real robotic platform demonstrate consistent improvements.For example, our method improves RDT-1B by 8\% on Robotwin 2.0, and boosts $\pi_0$ by 5.1\% on the more challenging LIBERO-Plus benchmark. Code is available at: https://github.com/nanfangxiansheng/Seeing-Realism-from-Simulation
vision-language-actiondata augmentationvideo generationdomain transfer
OpenReview ↗
PDF ↗
Code ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Self-CriTeach: LLM Self-Teaching and Self-Critiquing for Improving Robotic Planning via Automated Domain Generation
Jinbang Huang, Zhiyuan Li, Yuanzhao Hu, Zhanguang Zhang, Mark Coates, Xingyue Quan, Yingxue Zhang
Large Language Models (LLMs) have recently shown strong promise for robotic task planning, particularly through automatic planning domain generation. However, prior approaches largely treat generated planning domains as planning utilities, which are brittle under imperfect logical states and perception noise, overlooking their potential as scalable sources of reasoning supervision and structured reward signals. At the same time, reasoning LLMs depend on chain-of-thought (CoT) supervision that is expensive to collect for robotic tasks, and reinforcement learning (RL) faces challenges on reward engineering. We propose Self-CriTeach, an LLM self-teaching and self-critiquing framework in which an LLM autonomously generates symbolic planning domains that serve a dual role: (i) enabling large-scale generation of robotic planning problem–plan pairs, and (ii) providing structured reward functions. First, the self-written domains enable large-scale generation of symbolic task plans, which are automatically transformed into extended CoT trajectories for supervised fine-tuning. Second, the self-written domains are reused as structured reward functions, providing dense feedback for reinforcement learning without manual reward engineering. This unified training pipeline yields a planning-enhanced LLM with higher planning success rates, stronger cross-task generalization, reduced inference cost, and resistance to imperfect logical states.
robot task planningPDDLLLM planningreinforcement learning
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Sentinel-VLA: A Metacognitive VLA Model with Active Status Monitoring for Dynamic Reasoning and Error Recovery
Wenhao Li, Xiu Su, Dan Niu, Yichao Cao, Hongyan Xu, Zhe Qu, Lei Fan, Shan You, Chang Xu
Vision-language-action (VLA) models have advanced the field of embodied manipulation by harnessing broad world knowledge and strong generalization. However, current VLA models still face several key challenges, including limited reasoning capability, lack of status monitoring, and difficulty in self-correction. In this paper, we introduce \textbf{Sentinel-VLA}, a metacognitive VLA model equipped with an active ``sentinel'' module to monitor real-time execution status. Only when necessary, such as during initial planning or upon detecting an error, the model triggers a dynamic reasoning or formulate error recovery solutions. This on-demand reasoning mechanism ensures robust decision-making while minimizing computational overhead. Notably, all training data (spanning 44 tasks and over 2.6 million transitions) is automatically generated and annotated through our designed pipeline. We also propose the Self-Evolving Continual Learning (SECL) algorithm, which allows Sentinel-VLA to identify its capability boundaries and automatically collect data for expansion, paired with Orthogonal Continual Adapter (OC-Adapter) to constrain parameter updates to an orthogonal space, thereby preventing catastrophic forgetting. Real-world experiments demonstrate that Sentinel-VLA boosts the task success rate by over 30\% compared to the SOTA model, PI0. We will open-source all the code, weights, and data generation pipeline.
vision-language-actionerror recoverymetacognitionmanipulation
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 SG2Loc: Sequential Visual Localization on 3D Scene Graphs
Nicole Damblon, Olga Vysotska, Federico Tombari, Marc Pollefeys, Daniel Barath
Visual localization in complex indoor environments remains a critical challenge for robotics and AR applications. Sequential localization, where pose estimates are refined over time, is important for autonomous agents. However, traditional methods often require storing extensive image databases or point clouds, leading to significant overhead. This paper introduces a novel, lightweight approach to sequential visual localization using 3D scene graphs. Our method represents the environment with a compact scene graph, where nodes represent objects (with coarse meshes) and edges encode spatial relationships. For each image in the localization phase, we extract per-patch semantic features, predicting object identities. Localization is performed within a particle filter framework. Each particle, representing a camera pose, projects the coarse object meshes from the scene graph into the image, assigning object identities to patches based on visibility. The similarity of the per-patch features, in the input image, and object features from the scene graph determines the weight of a particle. Subsequent images are incorporated sequentially, refining the pose estimate. By leveraging a compact scene graph and efficient semantic matching, our method significantly reduces storage while maintaining performance on real-world datasets. The code is available at https://github.com/DmblnNicole/sg2loc.
visual-localizationscene-graphssequential-localizationslam
OpenReview ↗
PDF ↗
Code ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 SkillNet: Hierarchical Skill Modeling for Compositional Generalization in Vision-Language Action Models
Senwei Xie, Yuntian Zhang, Zhenzhou Tan, Ruiping Wang, Pengwei Wang, Shanghang Zhang, Xilin CHEN
Transfer across diverse task compositions and unseen behaviors remains a significant challenge for vision-language action (VLA) models. Skills are repeatable and atomic components for various tasks, and similarities shared with different skills provide evidence for transferability across behaviors. However, existing skill-centric methods have two problems. First, skills are often loosely organized, lacking a hierarchy that can capture similarities and differences across skills. Second, they lack a mechanism which has the capacity to express transferable skill attributes in a structured parametric space. To this end, we propose SkillNet, which models skill attributes in a hierarchical way and regulates compositional model structure with transferable skill attributes. SkillNet exploits motion code and VerbNet Framework to explicitly model similarities of skills on mechanical properties and semantic roles, and organizes skills in a hierarchical way. Based on this hierarchy, SkillNet leverages the scalability of the mixture-of-experts (MoE) mechanism and develops skill embeddings as soft constraints to enable compositional generalization via similar expert activations on similar skills. On zero-shot and few-shot transfer experiments in simulators and real-world environments, SkillNet achieves an improvement of performance by 16.0% and 23.9%. Meanwhile, SkillNet achieves state-of-the-art performance on in-domain settings.
hierarchical skillscompositional generalizationmixture-of-expertsmanipulation
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 SoMA: A Real-to-Sim Neural Simulator for Robotic Soft-Body Manipulation
Mu Huang, Hui Wang, Kerui Ren, Linning Xu, Yunsong Zhou, Mulin Yu, Bo Dai, Jiangmiao Pang
Simulating deformable objects under rich interactions remains a fundamental challenge for real-to-sim robot manipulation, with dynamics jointly driven by environmental effects and robot actions.
Existing simulators rely on predefined physics or data-driven dynamics without robot-conditioned control, limiting accuracy, stability, and generalization.
This paper presents \textbf{SoMA}, a 3D Gaussian Splat simulator for soft-body manipulation.
SoMA couples deformable dynamics, environmental forces, and robot joint actions in a unified latent neural space for end-to-end real-to-sim simulation.
Modeling interactions over learned Gaussian splats enables controllable, stable long-horizon manipulation and generalization beyond observed trajectories without predefined physical models.
SoMA improves resimulation accuracy and generalization on real-world robot manipulation by 20\%, enabling stable simulation of complex tasks such as long-horizon cloth folding. **Project Page: city-super.github.io/SoMA**
real-to-simsoft-body manipulationgaussian splattingdeformable objects
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 SpaCeFormer: Fast Proposal-Free Open-Vocabulary 3D Instance Segmentation
Chris Choy, Junha Lee, Chunghyun Park, Minsu Cho, Jan Kautz
Open-vocabulary 3D instance segmentation is a core capability for robotics and AR/VR, but prior methods trade one bottleneck for another: multi-stage 2D+3D pipelines aggregate foundation-model outputs at hundreds of seconds per scene, while pseudo-labeled end-to-end approaches rely on fragmented masks and external region proposals. We present SpaCeFormer, a proposal-free space-curve transformer that runs in 0.12–0.30 seconds per scene across standard benchmarks, 2–3 orders of magnitude faster than multi-stage 2D+3D pipelines. We pair it with SpaCeFormer-3M, the largest open-vocabulary 3D instance segmentation dataset (3.0M multi-view-consistent captions over 604K instances from 7.4K scenes) built through multi-view mask clustering and multi-view VLM captioning; it reaches **21× higher mask recall** than prior single-view pipelines (54.3% vs 2.5% at IoU>0.5). SpaCeFormer combines spatial window attention with Morton-curve serialization for spatially coherent features, and uses a RoPE-enhanced decoder to predict instance masks directly from learned queries without external proposals. On ScanNet200 we achieve 11.1 zero-shot mAP, a 2.8× improvement over the prior best proposal-free method; on ScanNet++ and Replica, we reach 22.9 and 24.1 mAP, surpassing all prior methods including those using multi-view 2D inputs.
3d-instance-segmentationopen-vocabularypoint-cloudscene-understanding
Sparse ActionGen: Accelerating Diffusion Policy with Real-time Pruning
Kangye Ji, Jianbo Zhou, Yuan Meng, Ye Li, Hanyun Cui, Zhi Wang
Diffusion Policy has dominated action generation due to its strong capabilities for modeling multi-modal action distributions, but its multi-step denoising processes make it impractical for real-time visuomotor control.
Existing caching-based acceleration methods typically rely on $\textit{static}$ schedules that fail to adapt to the \textit{dynamics} of robot-environment interactions, thereby leading to suboptimal performance.
In this paper, we propose $\underline{\textbf{S}}$parse $\underline{\textbf{A}}$ction$\underline{\textbf{G}}$en $(\textbf{SAG})$ for extremely sparse action generation. To accommodate the iterative interactions, SAG customizes a rollout-adaptive prune-then-reuse mechanism that first identifies prunable computations globally and then reuses cached activations to substitute them during action diffusion. To capture the rollout dynamics, SAG parameterizes an observation-conditioned diffusion pruner for environment-aware adaptation and instantiates it with a highly parameter- and inference-efficient design for real-time prediction. Furthermore, SAG introduces a one-for-all reusing strategy that reuses activations across both timesteps and blocks in a zig-zag manner, minimizing the global redundancy. Extensive experiments on multiple robotic benchmarks demonstrate that SAG achieves up to 4$\times$ generation speedup without sacrificing performance. Project Page: https://sparse-actiongen.github.io/.
diffusion policyreal-time controlmodel pruningvisuomotor policy
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Action
Pengteng Li, Weiyu Guo, He Zhang, Tiefu Cai, Xiao He, Yandong Guo, Hui Xiong
We introduce SOMA, the Spatial Memory framework for Out-of-Vision Manipulation in Vision-Language-Action (VLA) models. Most existing VLAs implicitly assume that task-relevant objects are always visible, leading to brittle and reactive behaviors when targets fall outside the camera’s field of view. SOMA addresses this limitation by equipping VLAs with a persistent spatial memory constructed from multi-view observations acquired via a movable head camera, enabling reasoning beyond the current visual frustum. The framework consists of three components: Spatial Memory Construction, which aggregates angular-wise observations into a unified spatial–semantic representation through scanning; Dynamic Memory Refinement, which maintains global consistency over time; and Contextual Memory Retrieval, which activates instruction-relevant spatial cues during manipulation. We evaluate SOMA on five challenging real-world out-of-vision manipulation tasks, including multi-step and dual-arm scenarios where target objects are initially invisible. Experimental results show that SOMA not only improves task success rates, but also induces qualitatively different manipulation behaviors, with faster target localization, reduced viewpoint search, and near one-shot grasping under partial observability. Additional experiments on RoboCasa GR1 and SimplerEnv further validate the effectiveness of SOMA’s memory design under conventional fully observable settings. Code will be released soon.
spatial memoryvision-language-actionpartial observabilitymanipulation
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 SpatioLM: Towards General Physical Spatial Intelligence in Vision-Language Models
jing wu, Jianhua Wu, Jiayi Guan, Jiahong Chen, Jinghui Lu, Hangjun Ye, Bingzhao Gao, Long Chen
Vision-Language Models (VLMs) perform well on commonsense reasoning tasks but struggle with visual spatial reasoning. Most existing solutions introduce extra 3D prior inputs or external spatial encoders, which increase complexity and degrade the underlying VLMs' general-purpose capabilities after spatial fine-tuning. To this end, we propose a parameter-efficient \textit{\textbf{Spatio}-vision \textbf{L}anguage \textbf{M}odels (SpatioLM)}, that enhances spatial intelligence without extra 3D prior inputs or third-party spatial encoders. Concretely, we design a plug-and-play and non-invasive spatio-vision module that elicits the spatial knowledge inherent in VLMs. Furthermore, we innovatively leverage pseudo depth and camera information as supervision to guide the model in learning physically coherent representations. Extensive experiments show that SpatioLM achieves significant improvements in diverse tasks, including spatial perception and understanding while effectively limiting the degradation of general capabilities. Notably, the model achieves an impressive score of 71.6 on the VSI-Bench (the first model to surpass 70). In addition, it attains competitive performance when transferred to embodied manipulation tasks.
vlmspatial-reasoningspatial-intelligencebenchmark
OpenReview ↗
PDF ↗
Poster Session 7 (Hall A) · Thursday, July 9, 2:30-4:15 SpecPrune-VLA: Accelerating Vision-Language-Action Models via Action-Aware Self-Speculative Pruning
Hanzhen Wang, Jiaming Xu, Yushun Xiang, Jiayi Pan, Yongkang Zhou, Yong-Lu Li, Guohao Dai
Pruning is a typical acceleration technique for compute-bound models by removing computation on unimportant values. Recently, it has been applied to accelerate Vision-Language-Action (VLA) model inference. However, existing acceleration methods focus on local information from the current action step and ignore the global context, leading to $>$20\% success rate drop and limited speedup in some scenarios. In this paper, we point out \textbf{spatial-temporal consistency} in VLA tasks: input images in consecutive steps exhibit high similarity, and propose the key insight that token selection should combine local information with global context of the model. Based on this, we propose ***SpecPrune-VLA***, a training-free, two-level pruning method with heuristic control.
**(1) Action-level static pruning.** We leverage global history and local attention to statically reduce visual tokens per action.
**(2) Layer-level dynamic pruning.** We prune tokens adaptively per layer based on layer-wise importance.
**(3) Lightweight action-aware controller:** We classify actions as coarse- or fine-grained by the speed of the end effector and adjust pruning aggressiveness accordingly. Extensive experiments show that SpecPrune-VLA achieves up to 1.57× speedup in LIBERO simulation and 1.70× on real-world tasks, with negligible success rate degradation.
vlapruninginference accelerationreal-world robots
OpenReview ↗
PDF ↗
Code ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Speedup Patch: Learning a Plug-and-Play Policy to Accelerate Embodied Manipulation
Zhichao Wu, Junyin Ye, Zhilong Zhang, Yihao Sun, Haoxin Lin, Jiaheng Luo, Haoxiang Ren, Lei Yuan, Yang Yu
While current embodied policies exhibit remarkable manipulation skills, their execution remains unsatisfactorily slow as they inherit the tardy pacing of human demonstrations. Existing acceleration methods typically require policy retraining or costly online interactions, limiting their scalability for large-scale foundation models. In this paper, we propose **S**peed**u**p **P**atch (**SuP**), a lightweight, policy-agnostic framework that enables **plug-and-play acceleration** using solely offline data. SuP introduces an external scheduler that adaptively downsamples action chunks provided by embodied policies to eliminate redundancies. Specifically, we formalize the optimization of our scheduler as a Constrained Markov Decision Process (CMDP) aimed at maximizing efficiency without compromising task performance. Since direct success evaluation is infeasible in offline settings, SuP introduces **World Model based state deviation** as a surrogate metric to enforce safety constraints. By leveraging a learned world model as a virtual evaluator to predict counterfactual trajectories, the scheduler can be optimized via offline reinforcement learning. Empirical results on simulation benchmarks (Libero, Bigym) and real-world tasks validate that SuP achieves an overall $1.8\times$ execution speedup for diverse policies while maintaining their original success rates.
manipulationpolicy accelerationoffline rlworld model
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 SPHERE: Mitigating the Loss of Spectral Plasticity in Mixture-of-Experts for Deep Reinforcement Learning
Lirui Luo, Guoxi Zhang, Hongming Xu, Cong Fang, Qing Li
In DRL, an agent is trained from a stream of experience. In a continual learning setting, such agents can suffer from \emph{plasticity loss}: their ability to learn new skills from new experiences diminishes over training. Recently, Mixture-of-Experts (MoE) networks have been reported to enable scaling laws and facilitate the learning of diverse skills. However, in continual reinforcement learning settings, their performance can degenerate as learning proceeds, indicating a loss of plasticity. To address this, building on Neural Tangent Kernel (NTK) theory, we formalize the plasticity loss in MoE policies as a loss of \emph{spectral plasticity}. We then derive a tractable proxy for spectral plasticity, one expressible in terms of individual expert feature matrices. Leveraging this proxy, we introduce \emph{SPHERE}, a practical Parseval penalty tailored for MoE-based policies that alleviates the loss of spectral plasticity. On MetaWorld and HumanoidBench, SPHERE improves average success under continual RL by 133\% and 50\% over an unregularized MoE baseline, while maintaining higher spectral plasticity throughout training.
reinforcement learningmixture-of-expertsplasticity lossmetaworldhumanoidbench
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 SpikeVLA: Vision-Language-Action Models with Spiking Neural Networks
Ruiqi Song, Dujun Nie, Siyu Teng, Baiyong Ding, Xiaotong Zhang, Dong Li, Chenming Zhang, Yuchen Li, Hangbin Wu, Long Chen
Vision-Language-Action (VLA) models have become a central paradigm for embodied intelligence. However, most existing approaches are built on large-scale Transformers, resulting in substantial inference latency and energy consumption that limit their practical deployment in low-power, real-time scenarios. We propose SpikeVLA, an end-to-end spiking VLA framework for embodied navigation with energy-efficient inference, consisting of three key components. (i) a spiking vision encoder, Spike-V, that replaces dense continuous computation with event-driven spiking representations to reduce the energy cost of visual representation learning, (ii) a multimodal spiking large language model, Spike-L, that reformulates cross-modal reasoning with spiking dynamics and token-level event-driven sparsity to further lower inference overhead, and (iii) a spiking action policy network, Spike-A, that uses Laplacian-kernel population coding and end-to-end reinforcement learning to produce stable, robust continuous control under low-energy constraints.
Experiments on multimodal interaction and robotic control tasks show that SpikeVLA significantly reduces energy consumption and computational overhead while maintaining competitive performance, highlighting its potential for low-power, real-time embodied intelligence.
vlaspiking neural networkembodied navigationenergy-efficient inference
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 STABLE: Simulation-Ready Tabletop Layout Generation via a Semantics–Physics Dual System
Zhen Luo, Yixuan Yang, Xudong XU, Jinkun Hao, Zhaoyang Lyu, Feng Zheng, Jiangmiao Pang, Yanwei Fu
Generating simulation-ready tabletop scenes from task instructions is an intriguing and promising research direction in the field of Embodied AI.
However, existing task-to-scene generation methods rely exclusively on large language models (LLMs) to predict scene layouts, inevitably yielding object collisions or floating due to LLMs’ inherent limitations in 3D spatial reasoning.
In this paper, we present \textbf{STABLE}, a semantics–physics dual-system tailored for simulation-ready tabletop scene generation.
STABLE consists of two complementary modules: (i) a \textbf{Semantic Reasoner}, a fine-tuned LLM trained on a structured tabletop scene dataset to generate coarse layouts from input task instructions, and (ii) a \textbf{Physics Corrector}, a physics-aware flow-based denoising model that outputs pose updates to refine layouts, which ensures the physical plausibility of scenes while preserves semantic alignment with task instructions.
STABLE adopts a progressive generation paradigm: by alternating between the Semantic Reasoner and Physics Corrector, it incrementally expands the scene from task-critical objects to background objects.
Experiments demonstrate that STABLE successfully generates simulation-ready tabletop scenes that strictly conform to task instructions and significantly enhances the physical validity of scenes over prior art.
scene generationtabletop manipulationsimulation-readyembodied ai
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 7 (Hall A) · Thursday, July 9, 2:30-4:15 StableVLA: Towards Robust Vision-Language-Action Models without Extra Data
YIYANG FU, Chubin Zhang, Shukai Gong, Yufan Deng, Kaiwei Sun, Qiyang Min, Qibin Hou, Yansong Tang, Jianan Wang, Daquan Zhou
It is infeasible to encompass all possible disturbances within the training dataset. This raises a critical question regarding the robustness of Vision-Language-Action (VLA) models when encountering unseen real-world visual disturbances, particularly under imperfect visual conditions. In this work, we conduct a systematic study based on recent state-of-the-art VLA models and reveal a significant performance drop when visual disturbances absent from the training data are introduced. To mitigate this issue, we propose a lightweight adapter module grounded in information theory, termed the Information Bottleneck Adapter (IB-Adapter), which selectively filters potential noise from visual inputs. Without requiring any extra data or augmentation strategies, IB-Adapter consistently improves over the baseline by an average of 30%, while adding fewer than 10M parameters, demonstrating notable efficiency and effectiveness. Furthermore, even with a 14x smaller backbone (0.5B parameters) and no pre-training on the Open X-Embodiment dataset, our model StableVLA achieves robustness competitive with 7B-scale state-of-the-art VLAs. With negligible parameter overhead (<10M), our approach maintains accuracy on long-horizon tasks and surpasses OpenPi under real-world visual disturbances. The code will be made publicly available.
vlarobustnessinformation bottleneckreal-world
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 STEP: Warm-Started Visuomotor Policies with Spatiotemporal Consistency Prediction
Jinhao Li, Yuxuan Cong, Yingqiao Wang, Hao Xia, Shan Huang, Yijia Zhang, Ningyi Xu, Guohao Dai
Diffusion policies have recently emerged as a powerful paradigm for visuomotor control in robotic manipulation due to their ability to model the distribution of action sequences and capture multimodality. However, iterative denoising leads to substantial inference latency, limiting control frequency in real-time closed-loop systems. Existing acceleration methods either reduce sampling steps, bypass diffusion through direct prediction, or reuse past actions, but often struggle to jointly preserve action quality and achieve consistently low latency.
In this work, we propose **STEP**, a lightweight spatiotemporal consistency prediction mechanism to construct high-quality warm-start actions that are both distributionally close to the target action and temporally consistent, without compromising the generative capability of the original diffusion policy.
Then, we propose a velocity-aware perturbation injection mechanism that adaptively modulates actuation excitation based on temporal action variation to prevent execution stall especially for real-world tasks.
We further provide a theoretical analysis showing that the proposed prediction induces a locally contractive mapping, ensuring convergence of action errors during diffusion refinement.
Extensive evaluations on nine simulated benchmarks and two real-world tasks show that STEP with 2 steps can achieve an average 21.6\% and 27.5\% higher success rate than BRIDGER and DDIM on the RoboMimic benchmark and real-world tasks, respectively.
The code is available at https://github.com/Kimho666/STEP.
diffusion policyvisuomotor controlinference accelerationreal-world robots
OpenReview ↗
PDF ↗
Code ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Structured 4D Latent Predictive Model for Robot Planning
Zhiyi Li, Peilin Wu, Xiaoshen Han, Ruojin Cai, Yilun Du
Video predictive models are emerging as a powerful paradigm in robotics, offering a promising path toward task generalization, long-horizon planning, and flexible decision-making. However, prevailing approaches often operate on 2D video sequences, inherently lacking the 3D geometric understanding necessary for precise spatial reasoning and physical consistency. We introduce a **Structured 4D Latent Predictive Model**, which predicts the evolution of a scene’s 3D structure in a structured latent space conditioned on observations and textual instructions. Our representation encodes the scene holistically and can be decoded into diverse 3D formats, enabling a more complete and 3D consistent scene understanding. This structured 4D latent predictive model serves as a planner, generating future scenes that are translated into executable actions by a goal-conditioned inverse dynamics module. Experiments demonstrate that our model generates futures with strong visual quality, substantially better 3D consistency and multi-view coherence compared to state-of-the-art video-based planners. Consequently, our full planning pipeline achieves superior performance on complex manipulation tasks, exhibits robust generalization to novel visual conditions, and proves effective on real-world robotic platforms. Our website is available at https://structured-4d-model.github.io/.
world model4d predictionrobot planningmanipulation
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Tabero: Learning Gentle Manipulation with Closed-Loop Force Feedback from Vision, Touch, and Language
Qiwei Wu, Rui Zhang, Xin Xiang, Tao Li, Weihua Zhang, Junjie Lai, Renjing Xu
Tactile sensing is essential for robots to achieve human-like gentle manipulation. However, existing Vision-Language-Action (VLA) models struggle to exploit tactile feedback for gentle manipulation due to scarce aligned vision-tactile-language data and the lack of effective closed-loop force feedback mechanisms.
To address these challenges, we introduce Tabero, a benchmark and model suite for gentle, language-conditioned robotic manipulation that demands fine-grained contact force perception. First, the Tabero benchmark addresses the scarcity of tactile data by presenting a data-efficient pipeline that repurposes open-source robot manipulation trajectories to generate diverse vision-tactile-language tasks, and establishes a multidimensional evaluation protocol that measures task success alongside physical interaction quality. Second, we propose Tabero-VTLA, an architecture with a decoupled force-position command interface; the resulting force-position commands are executed by a fixed hybrid controller to enable real-time, force-aware manipulation. Evaluated on Tabero, our model maintains high task success while reducing average grip force by over 70\% under gentle instructions, demonstrating its ability to modulate interaction forces based on multimodal experience.
tactile sensingforce feedbackvlagentle manipulation
OpenReview ↗
PDF ↗
Code ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 TapSampling: Inference-Time Sampling with a Task-Progress-Understanding Verifier for Robotic Manipulation
Sizhe Zhao, Shengping Zhang, Shuo Yang, Weiyu Zhao, Shuigen Wang, Xiangyang Ji
Existing embodied control research demonstrates remarkable performance improvements by scaling training data and model size.
We instead explore inference-time strategy as an alternative axis. Non-deterministic generative models, such as diffusion and autoregressive models, have been widely adopted in the field of embodied control. However, the single-shot inference paradigm limits their performance. In this paper, we propose **TapSampling**, a plug-and-play framework for inference-time sampling. First, we introduce an Action-VAE that represents actions in a low-dimensional latent space by mapping policy-generated initial actions into a compressed posterior distribution, from which any number of latent samples can be drawn and decoded into candidate actions that approximate the true action distribution. Second, we formulate action verification as task-progress outcome prediction, using the intrinsic sequential structure of robotic datasets to train a semantically grounded verifier for interpretable action selection. Furthermore, TapSampling is a policy-agnostic framework. Extensive experiments in both simulated and real-world environments demonstrate that our method substantially improves multiple generalist policies without further policy finetuning. Code and models are available at the **project page** (https://aipixel.github.io/TapSampling/).
inference-time samplingembodied aigeneralist policiesaction verification
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
Shelly Francis-Meretzki, Mirco Mutti, Yaniv Romano, Aviv Tamar
Recent advances in vision-language-action (VLA) models for robotics have highlighted the importance of reliable uncertainty quantification in sequential tasks. However, assessing and improving calibration in such settings remains mostly unexplored, especially when only partial trajectories are observed. In this work, we formulate *sequential calibration* for episodic tasks, where task-success confidence is produced along an episode, while success is determined at the end of it. We introduce a sequential extension of the Brier score and show that, for binary outcomes, its risk minimizer coincides with the VLA policy’s value function. This connection bridges uncertainty calibration and reinforcement learning, enabling the use of temporal-difference (TD) value estimation as a principled calibration mechanism over time.
We empirically show that TD calibration improves performance relative to the state-of-the-art on simulated and real-robot data. Interestingly, we show that when calibrated using TD, the VLA's single-step action probabilities can yield competitive uncertainty estimates, in contrast to recent findings that employed different calibration techniques.
vlacalibrationuncertaintysafetyreal-robot
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 1 (Hall A) · Tuesday, July 7, 10:30-12:15 Test-time Offline Reinforcement Learning on Goal-related Experience
Marco Bagatella, Mert Albaba, Jonas Hübotter, Georg Martius, Andreas Krause
Foundation models compress a large amount of information in a single, large neural network, which can then be queried for individual tasks. There are strong parallels between this widespread framework and offline goal-conditioned reinforcement learning algorithms: a universal value function is trained on a large number of goals, and the policy is evaluated on a single goal in each test episode. Extensive research in foundation models has shown that performance can be substantially improved through test-time training, specializing the model to the current goal. We find similarly that test-time offline reinforcement learning on experience related to the test goal can lead to substantially better policies at modest compute costs. We propose a novel self-supervised data selection criterion, which selects transitions from an offline dataset according to their relevance to the current state and quality with respect to the evaluation goal. We demonstrate across a wide range of high-dimensional loco-navigation and manipulation tasks that fine-tuning a policy on the selected data for a few gradient steps leads to significant performance gains over standard offline pre-training. Our goal-conditioned test-time training (GC-TTT) algorithm applies this routine in a receding-horizon fashion during evaluation, adapting the policy to the current trajectory as it is being rolled out. Finally, we study compute allocation at inference, demonstrating that, at comparable costs, GC-TTT induces performance gains that are not achievable by scaling model size.
offline rlgoal-conditioned rltest-time traininglocomotionmanipulation
OpenReview ↗
PDF ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 TF-FACE: Time-Frequency Fusion Learning via Frequency-Domain Adaptive and Controllable Enhancement for Trajectory Prediction
Dongjian Song, Yunhao Meng, Songjun Huang, Jiayi Han
Accurately predicting the future trajectories of traffic participants is critical for safe, efficient, and human-friendly autonomous driving. Existing learning-based trajectory prediction methods are predominantly time-domain and insufficiently exploit latent frequency information, which limits their capability to capture low-frequency long-term dependencies and high-frequency short-term dynamics. To address this, we propose TF-FACE, a time-frequency learning framework via frequency-domain adaptive and controllable enhancement. TF-FACE introduces a fusion encoder with learnable gated frequency-domain attention that adaptively manipulates band-specific features for trajectory prediction. Building on the fused representation, we design a dual-stage decoder and a band-specific time–frequency dual-consistency loss to enable controllable decoupling and coupling across long- and short-term temporal scales, global and local scales, and then generate final multimodal predictions. Experiments on Argoverse 1 demonstrate that TF-FACE achieves state-of-the-art accuracy, while maintaining real-time inference for autonomous driving. Additional experiments are conducted on Argoverse 2, further validating TF-FACE's performance and generalizability.
trajectory-predictionautonomous-drivingtime-frequency-fusionperception
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 The Lie We Tell: Correcting the Euclidean Fallacy in Vision Language Action Policies via Score Matching on Tangent Space
Bing-Cheng Chuang, I-Hsuan Chu, Bor-Jiun Lin, YuanFu Yang, Min Sun, Chun-Yi Lee
Diffusion-based Vision-Language-Action policies achieve remarkable success in robotic manipulation, yet commit a fundamental geometric error we term the **Euclidean Fallacy**: representing SE(3) poses as flat $\mathbb{R}^{12}$ vectors. This approximation induces (1) manifold drift violating SO(3) constraints, (2) broken equivariance under coordinate transformations, and (3) non-geodesic trajectories with excessive kinematic cost. We introduce **Lie Diffuser Actor (LDA)**, a diffusion framework operating intrinsically on SE(3). Our method injects noise through left-invariant SDEs, predicts scores in the tangent space, and retracts samples via the exponential map. This formulation eliminates manifold drift by construction while guaranteeing coordinate-frame equivariance and geodesic optimality. On CALVIN ABC$\rightarrow$D, LDA improves average task length from $3.27$ to $3.51$ ($+7.3\%$). We further validate our method on real robot and the results show that our methodology outperforms the baseline on majority tasks.
vlase(3) posediffusion policymanipulationlie groups
OpenReview ↗
PDF ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 Think Less, Act Early: Reinforced Latent Reasoning with Early Exit in Vision-Language-Action Models
Dianqiao Lei, Lianlei Shan
Existing Vision-Language-Action (VLA) models predominantly rely on explicit Chain-of-Thought (CoT) reasoning to bridge perception and action. While effective, this paradigm suffers from high computational costs and error propagation in multi-step tasks. In this paper, we propose Adaptive Variable Alignment VLA (AVA-VLA), a novel Latent Reasoning VLA framework that models reasoning as a sequence of unobservable latent variables, bypassing the need for explicit text generation. However, latent trajectories are inherently susceptible to noise interference and misalignment with downstream objectives. To address this, we introduce a Reinforcement Learning-based Denoising mechanism that treats latent state generation as a sequential decision process, optimizing reasoning trajectories via task-level rewards. Furthermore, we incorporate an Early-Exit Strategy that adaptively terminates reasoning based on state confidence, enabling a dynamic trade-off between depth and efficiency. Extensive experiments on embodied decision benchmarks demonstrate that AVA-VLA significantly reduces inference latency while achieving superior stability and success rates compared to full-reasoning baselines.
vlalatent reasoningearly exitreinforcement learningembodied ai
OpenReview ↗
PDF ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 Thinking with Geometry: Active Geometry Integration for Spatial Reasoning
Haoyuan Li, Qihang Cao, Tao Tang, Kun Xiang, Zihan Guo, Jianhua Han, Jia-Wang Bian, Hang Xu, Xiaodan Liang
Recent progress in spatial reasoning with Multimodal Large Language Models (MLLMs) increasingly leverages geometric priors from 3D encoders. However, most existing integration strategies remain passive: geometry is exposed as a global stream and fused in an indiscriminate manner, which often induces semantic-geometry misalignment and redundant signals. We propose GeoThinker, a framework that shifts the paradigm from passive fusion to active perception. Instead of feature mixing, GeoThinker enables the model to selectively retrieve geometric evidence conditioned on its internal reasoning demands. GeoThinker achieves this through Spatial-Grounded Fusion applied at carefully selected VLM layers, where semantic visual priors selectively query and integrate task-relevant geometry via frame-strict cross-attention, further calibrated by Importance Gating that biases per-frame attention toward task-relevant structures. Comprehensive evaluation results show that GeoThinker sets a new state-of-the-art in spatial intelligence, achieving a peak score of 72.6 on the VSI-Bench. Furthermore, GeoThinker demonstrates robust generalization and significantly improved spatial perception across complex downstream scenarios, including embodied referring and autonomous driving. Our results indicate that the ability to actively integrate spatial structures is essential for next-generation spatial intelligence.
mllmspatial-reasoninggeometrymultimodal
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 7 (Hall A) · Thursday, July 9, 2:30-4:15 TIC-VLA: A Think-in-Control Vision-Language-Action Model for Robot Navigation in Dynamic Environments
Zhiyu Huang, Yun Zhang, Johnson Liu, Rui Song, Chen Tang, Jiaqi Ma
Robots in dynamic, human-centric environments must follow language instructions while maintaining real-time reactive control. Vision-language-action (VLA) models offer a promising framework, but they assume temporally aligned reasoning and control, despite semantic inference being inherently delayed relative to real-time action. We introduce Think-in-Control (TIC)-VLA, a latency-aware framework that explicitly models delayed semantic reasoning during action generation. TIC-VLA defines a delayed semantic-control interface that conditions action generation on delayed vision-language semantic states and explicit latency metadata, in addition to current observations. We further propose a latency-consistent training pipeline that injects reasoning inference delays during imitation learning and online reinforcement learning, aligning training with asynchronous deployment. To support realistic evaluation, we present DynaNav, a physics-accurate, photo-realistic simulation suite for language-guided navigation in dynamic environments. Extensive experiments in simulation and on a real robot show that TIC-VLA consistently outperforms prior VLA models while maintaining robust real-time control under multi-second reasoning latency. Project website: https://ucla-mobility.github.io/TIC-VLA/
vlarobot navigationlatency-aware controlreal-robot
TimeRewarder: Learning Dense Reward from Passive Videos via Frame-wise Temporal Distance
Yuyang Liu, Chuan Wen, Yihang Hu, Dinesh Jayaraman, Yang Gao
Designing dense rewards is crucial for reinforcement learning (RL), yet in robotics it often demands extensive manual effort and lacks scalability. One promising solution is to view task progress as a dense reward signal, as it quantifies the degree to which actions advance the system toward task completion over time. We present TimeRewarder, a simple yet effective reward learning method that derives progress estimation signals from passive videos, including robot demonstrations and human videos, by modeling temporal distances between frame pairs. We then demonstrate how TimeRewarder can supply step-wise proxy rewards to guide reinforcement learning. In our comprehensive experiments on ten challenging Meta-World tasks, we show that TimeRewarder dramatically improves RL for sparse-reward tasks, achieving nearly perfect success in 9/10 tasks with only 200,000 interactions per task with the environment. This approach outperforms previous methods and even the manually designed environment dense reward on both the final success rate and sample efficiency. Moreover, we show that TimeRewarder pretraining can exploit real-world human videos, highlighting its potential as a scalable approach to rich reward signals from diverse video sources.
reward learningrobotic manipulationvideo-based rewardmeta-world
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 8 (Hall A) · Thursday, July 9, 5:00-6:45 Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning
Sungyoung Lee, Dohyeong Kim, Eshan Balachandar, Zelal Su Mustafaoglu, Keshav Pingali
We propose Flow-Anchored Noise-conditioned Q-Learning (FAN), a highly efficient and high-performing offline reinforcement learning (RL) algorithm. Recent work has shown that expressive flow policies and distributional critics improve offline RL performance, but at a high computational cost. Specifically, flow policies require iterative sampling to produce a single action, and distributional critics require computation over multiple samples (e.g., quantiles) to estimate value. To address these inefficiencies while maintaining high performance, we introduce FAN. Our method employs a behavior regularization technique that uses a single flow policy iteration and requires a single Gaussian noise sample for distributional critics. Our theoretical analysis of convergence and performance bounds demonstrates that these simplifications not only improve efficiency but also lead to superior task performance. Experiments on robotic manipulation and locomotion tasks demonstrate that FAN achieves state-of-the-art performance while significantly reducing both training and inference runtimes. We release our code at https://github.com/brianlsy98/FAN.
offline rlflow policyrobotic manipulationlocomotion
OpenReview ↗
PDF ↗
Code ↗
Project page ↗
Poster Session 7 (Hall A) · Thursday, July 9, 2:30-4:15 Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models
Zhilong Zhang, Haoxiang Ren, Yihao Sun, Yifei Sheng, Haonan Wang, Zhichao Wu, Haoxin Lin, Pierre-Luc Bacon, Yang Yu
Vision-Language-Action (VLA) models show strong generalization for robotic control, but finetuning them with reinforcement learning (RL) is constrained by the high cost and safety risks of real-world interaction. Training VLA models in interactive world models avoids these issues but introduces several challenges, including pixel-level world modeling, multi-view consistency, and compounding errors under sparse rewards. Building on recent advances across large multimodal models and model-based RL, we propose VLA-MBPO, a practical framework to tackle these problems in VLA finetuning. Our approach has three key design choices: (i) adapting unified multimodal models (UMMs) for data-efficient world modeling; (ii) an interleaved view decoding mechanism to enforce multi-view consistency; and (iii) chunk-level branched rollout to mitigate error compounding. Theoretical analysis and experiments across simulation and real-world tasks demonstrate that VLA-MBPO significantly improves policy performance and sample efficiency. Crucially, our method maintains a universal set of hyperparameters across all tasks, underscoring its robustness and scalability for real-world robotic deployment.
vlaworld modelmodel-based rlreal-world robots
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Training-Free Guided Diffusion for Planning: A Unified Framework via Doob’s h-Transform with Safety Guarantees
Kenta Hoshino, Yashaswi Shashank Aluru, Xiyu Deng, Yorie Nakahira
This paper studies the theoretical foundations of guidance mechanisms in continuous-time score-based diffusion models.
We adopt Doob’s h-transform as a principled framework for characterizing ideal guided diffusion processes and analyze the discrepancy between ideal and approximate guidance.
Our analysis provides explicit error bounds and yields probabilistic guarantees on satisfying prescribed constraints, which are particularly important for safety-critical planning.
We further show that the Doob-based formulation induces a stochastic optimal control problem, enabling practical guidance design without additional model training.
We demonstrate the effectiveness of the proposed framework on robotic navigation tasks, including language-conditioned planning.
diffusion planningsafety guaranteesrobot navigationguided diffusion
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Transferable Reinforcement Learning via Probabilistic Latent Embeddings and Dynamic Policy Adaptation for Sim-to-Real Deployment
Gengyue Han, Yiheng Feng
Due to limited resources and public safety concerns, deep reinforcement learning (RL) agents for many cyber-physical systems (e.g., autonomous vehicles) are first trained in simulators. However, when deployed in real world environments, they often suffer from performance degradation or safety violations because of the inevitable Sim2Real gap. Existing zero-shot approaches, such as robust safe RL and domain randomization, mitigate this issue but typically at the cost of degraded performance or residual safety risks when experiencing unmodeled system dynamics. To address these limitations, we propose a novel reinforcement learning framework that enables safe and efficient policy transfer via probabilistic latent embeddings and dynamic policy adaptation. We consider a family of Constrained Markov Decision Processes (CMDPs) under different environment contexts. By leveraging latent context variable in meta-RL, the proposed framework infers the latent representation of the environment from simulated experiences. Furthermore, it incorporates a distributional RL formulation, which allows risk levels of the deployed policy to be adjusted dynamically, based on the estimation accuracy of the latent context variable. This strategy promotes safety at the early deployment stage and improves efficiency through fast policy adaptation under the Sim2Real gap.
sim2realmeta-rlsafe rlautonomous vehiclesdistributional rl
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 TransNormal: Dense Visual Semantics for Diffusion-based Transparent Object Normal Estimation
Mingwei Li, Hehe Fan, Yi Yang
Monocular normal estimation for transparent objects is critical for laboratory automation, yet it remains challenging due to complex light refraction and reflection. These optical properties often lead to catastrophic failures in conventional depth and normal sensors, hindering the deployment of embodied AI in scientific environments. We propose **TransNormal**, a novel framework that adapts pre-trained diffusion priors for single-step normal regression. To handle the lack of texture in transparent surfaces, TransNormal integrates dense visual semantics from DINOv3 via a cross-attention mechanism, providing strong geometric cues. Furthermore, we employ a multi-task learning objective and wavelet-based regularization to ensure the preservation of fine-grained structural details. To support this task, we introduce **TransNormal-Synthetic**, a physics-based dataset with high-fidelity normal maps for transparent labware. Extensive experiments demonstrate that TransNormal significantly outperforms state-of-the-art methods: on the ClearGrasp benchmark, it reduces mean error by 25.5\% and improves the best prior $11.25^\circ$ accuracy by 24.7\%; on ClearPose, it achieves a 17.7\% reduction in mean error. Code and dataset are publicly available at https://github.com/longxiang-ai/TransNormal.
normal-estimationtransparent-objectsperceptiondiffusion-model
OpenReview ↗
PDF ↗
Code ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 TRAP: Hijacking VLA CoT-Reasoning via Adversarial Patches
Zhengxian Huang, Wenjun Zhu, Haoxuan Qiu, Xiaoyu Ji, Wenyuan Xu
By integrating Chain-of-Thought (CoT) reasoning, Vision-Language-Action (VLA) models have demonstrated strong capabilities in robotic manipulation, particularly by improving generalization and interpretability. However, the security of CoT-based reasoning mechanisms remains largely unexplored. In this paper, we show that CoT reasoning introduces a novel attack vector for targeted behavior hijacking—for example, causing a robot to mistakenly deliver a knife to a person instead of an apple—without modifying the user’s instruction. We first provide empirical evidence that CoT strongly governs action generation, even when it is semantically misaligned with the input instructions. Building on this observation, we propose **TRAP**, the first targeted behavior-hijacking adversarial attack against CoT-reasoning VLA models. By targeting the reasoning-to-action pathway, **TRAP** uses an adversarial patch (*e.g.*, a tablecloth placed on the table) to steer intermediate CoT reasoning and downstream actions toward adversary-defined behaviors. Extensive evaluations on three representative reasoning VLAs, spanning distinct CoT reasoning mechanisms, demonstrate the effectiveness of **TRAP**. Notably, we implemented the patch by printing it on paper in a real-world setting. Our findings highlight the urgent need to secure CoT reasoning in VLA systems. The project page is available at https://zhengxian-huang.github.io/TRAP-website/.
vlaadversarial patchsafetyrobot manipulationchain-of-thought
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 8 (Hall A) · Thursday, July 9, 5:00-6:45 Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates
Anish Abhijit Diwan, Davide Tateo, Christopher Mower, Haitham Bou Ammar, Jan Peters, Oleg Arenz
Inverse reinforcement learning (IRL) is typically formulated as maximizing entropy subject to matching the distribution of expert trajectories. Classical (dual-ascent) IRL guarantees monotonic performance improvement but requires fully solving an RL problem each iteration to compute dual gradients. More recent adversarial methods avoid this cost at the expense of stability and monotonic dual improvement, by directly optimizing the primal problem and using a discriminator to provide rewards. In this work, we bridge the gap between these approaches by enabling monotonic improvement of the reward function and policy without having to fully solve an RL problem at every iteration. Our key theoretical insight is that a trust-region-optimal policy for a reward function update can be globally optimal for a smaller update in the same direction. This smaller update allows us to explicitly optimize the dual objective while only relying on a local search around the current policy. In doing so, our approach avoids the training instabilities of adversarial methods, offers monotonic performance improvement, and learns a reward function in the traditional sense of IRL—one that can be globally optimized to match expert demonstrations. Our proposed algorithm, *Trust Region Inverse Reinforcement Learning (TRIRL)*, outperforms state-of-the-art imitation learning methods across multiple challenging tasks by a factor of 2.4x in terms of aggregate inter-quartile mean, while recovering reward functions that generalize to system dynamics shifts.
inverse-rlimitation-learningtrust-regionreward-learning
Trust-Region Diffusion Policies for Massively Parallel On-Policy RL
Huy Le, Onur Celik, Denis Blessing, Tai Hoang, Claas A Voelcker, Axel Brunnbauer, Felix Richter, Michael Volpp, Gerhard Neumann
Reinforcement learning with massively parallel simulations has become a standard framework for developing robust, deployable policies; however, most existing approaches still rely on simple Gaussian policy parameterizations. Diffusion models provide a more expressive policy class and have shown strong performance on challenging control problems, yet most diffusion-based RL methods are designed for offline or off-policy training. In this work, we ask whether diffusion policies can be trained effectively in the massively parallel, on-policy regime. To this end, we introduce Trust-region Diffusion Policies (TruDi), which enables diffusion policies for on-policy RL with massively parallel simulations. This setting is particularly challenging because the data distribution changes quickly across updates, making stable training with complex policies difficult. TruDi addresses this by integrating a trust-region optimization rule to enforce a KL-divergence constraint over the entire diffusion trajectory. Empirically, we evaluate TruDi on a diverse set of 4 massively parallel RL benchmarks comprising a total of 73 tasks. Across these tasks, TruDi consistently outperforms or is on-par with strong baselines on standard tasks and achieves clear gains on more challenging humanoid control tasks, establishing a strong new baseline for massively parallel on-policy RL.
diffusion policyon-policy rlmassively parallel rlhumanoid control
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Turning Adaptation into Assets: Cross-Domain Bridging for Online Vision-Language Navigation
Zixuan Hu, Xuantuo Huang, Yancheng Li, Yichun Hu, Shengyong Xu, LINGYU DUAN
Navigating under non-stationary environment shifts poses a critical challenge for a Vision-and-Language Navigation (VLN) agent deployed in the wild. Yet, existing Test-Time Adaptation (TTA) methods for VLN largely treat online adaptation as transient, isolated updates, leading to catastrophic forgetting and negative transfer. To overcome these issues, we propose **I**nter-**D**omain Bridg**E** with Historical **A**ssets (**IDEA**), a novel TTA framework that transforms adaptation into the accumulation and composition of assets. Specifically, IDEA introduces soft prompts optimized via a Fisher-guided weighting scheme to capture the transferable knowledge. These optimized prompts are then augmented with domain coordinates to form a dynamic asset library. Leveraging this library, IDEA constructs a cross-domain bridge by projecting the target domain onto the convex hull of historical knowledge. These designs form a complementary loop: the evolving library underpins bridge construction, while the bridge provides superior initialization to accelerate asset optimization. Extensive experiments across REVERIE, R2R, and R2R-CE benchmarks demonstrate the consistent superiority of IDEA over existing methods, showcasing its ability to enable training-free adaptation via asset sharing.
vision-language navigationtest-time adaptationembodied agent
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 UAV$^2$: A Unified and Adaptive Scheduling Framework for UAV Autopilot System with Reinforcement Learning
Zeying Li, shuai zhao, Chaowen Wu, Boyang Li, Kai Huang
Unmanned aerial vehicle (UAV) autopilot systems typically comprise navigation and flight-control modules, and their effective scheduling is critical to achieving high flight performance.
However, most existing UAV platforms adopt a split architecture in which navigation and flight control are deployed on separate hardware devices. This separation restricts system-wide observability and prevents holistic scheduling and optimization across the entire autopilot pipeline.
Moreover, autonomous flight performance emerges from implicit, cross-coupled, and accumulated interactions among multiple factors, rendering traditional model-based or heuristic scheduling approaches ineffective.
To address these challenges, we propose UAV$^2$, a unified and adaptive scheduling framework for UAV autopilot systems with reinforcement learning, targeting flight performance optimization.
UAV$^2$ integrates navigation and flight control onto a single onboard computing platform and operating system, formulates the scheduling problem as a partially observable Markov decision process, and learns scheduling policies from runtime execution feedback.
The proposed approach is trained and evaluated in a hardware-in-the-loop simulation environment. Experimental results demonstrate that the learned scheduling policy consistently outperforms fixed-rate scheduling strategies in terms of flight robustness and tracking performance.
uavautopilotreinforcement learningflight controlscheduling
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Uncertainty-Guided Exploration and Stable Planning for Sparse-Reward Manipulation from Limited Demonstrations
Haowen Sun, Liqi Huang, Mingyang Li, Sihua Ren, Xinzhe Chen, Chengzhong Ma, Zeyang Liu, Xingyu Chen, Xuguang Lan
Reinforcement learning from demonstrations (RLfD) offers a promising method for robotic manipulation with sparse rewards. However, limited demonstrations often cause agents to encounter out-of-distribution states where world models produce poor predictions. In multi-stage tasks, jointly optimizing a learned reward function and policy introduces a moving target problem, and the resulting non-stationarity intensifies the impact of uncertainty on policy learning. In this work, we propose QUEST, a model-based RL framework that adaptively switches between exploration and exploitation guided by uncertainty to achieve stable and efficient learning. Specifically, our approach employs intrinsic rewards to encourage exploration, leverages ensemble dynamics for uncertainty-guided planning, and introduces a hybrid sampling strategy to prioritize rare successful stage transitions. We evaluate QUEST on challenging sparse-reward manipulation tasks with limited expert demonstrations. Results show that QUEST outperforms state-of-the-art methods by 17\% on average, with gains increasing to 60\% on difficult tasks. We further demonstrate successful zero-shot sim-to-real transfer on five real-world tasks. Project website: https://quest-official.github.io/QUEST/.
sparse-reward rlsim-to-realmanipulationuncertainty-guided exploration
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 Understanding Behavior Cloning with Action Quantization
Haoqun Cao, Tengyang Xie
Behavior cloning is a fundamental paradigm in machine learning, enabling policy learning from expert demonstrations across robotics, autonomous driving, and generative models. Autoregressive models like transformer have proven remarkably effective, from large language models (LLMs) to vision-language-action systems (VLAs). However, applying autoregressive models to continuous control requires discretizing actions through quantization, a practice widely adopted yet poorly understood theoretically.
This paper provides theoretical foundations for this practice. We analyze how quantization error propagates along the horizon and interacts with statistical sample complexity. We show that behavior cloning with quantized actions and log-loss achieves optimal sample complexity—matching existing lower bounds—and incurs only polynomial horizon dependence on quantization error, provided the dynamics are stable and the policy satisfies a probabilistic smoothness condition. We further characterize when different quantization schemes satisfy or violate these requirements, and propose a model-based augmentation that provably improves the error bound without requiring policy smoothness. Finally, we establish fundamental limits that jointly capture the effects of quantization error and statistical complexity.
behavior-cloningaction-quantizationtheorytokenization
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 UniJEPA: Enhancing Robot Policy via Unified Continuous and Discrete Representation Learning
Jianke Zhang, Yucheng Hu, Yanjiang Guo, Xiaoyu Chen, Yichen Liu, Wenna Chen, Chaochao Lu, Jianyu Chen
Building generalist robot policies that can handle diverse tasks in open-ended environments is a central challenge in robotics. To leverage knowledge from large-scale pretraining, prior work (VLA) has typically built generalist policies either on top of vision-language models (VLMs) or generative models. However, both semantic understanding from vision-language pretraining and visual dynamics modeling from visual-generation pretraining are crucial for embodied robots.
Recent unified models of generation and understanding have demonstrated strong capabilities in both comprehension and generation through large-scale pretraining. We posit that robotic policy learning can likewise benefit from the combined strengths of understanding, planning and continuous future representation learning. Building on this insight, we introduce UniJEPA, which acquires the ability to dynamically model high-dimensional visual features through pretraining on over 1M internet-scale instructional manipulation videos. Subsequently, UniJEPA is fine-tuned on data collected from the robot embodiment, enabling the learning of mappings from predictive representations to action tokens. Extensive experiments show our approach consistently outperforms baseline methods in terms of 9\% and 12\% across simulation environments and real-world out-of-distribution tasks.
robot policyrepresentation learningmanipulationvla
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 UniMapping: Unified SLAM Framework for Map-Centric Embodied Perception
Xiaze Zhang, Ziheng Ding, Yuejie Zhang, lifeng chen, Rui Feng
Simultaneous Localization and Mapping (SLAM) is increasingly expected to provide reusable spatial representations for downstream perception. However, existing approaches often struggle with scale-consistency and producing maps that lack the geometric fidelity required for reliable perception. We propose _UniMapping_, a unified SLAM framework that constructs a persistent neural-descriptor map from multimodal observations. We introduce a **Spatial-Aware Deformable Transformer** that injects explicit geometric inductive bias to ensure scale-invariant feature extraction, alongside a **Spatial Fusion** strategy that decouples feature aggregation from temporal sequences. Extensive experiments on both indoor and outdoor benchmarks demonstrate competitive SLAM performance. Notably, our method significantly enhances downstream tasks (mAP +3.1% and mIoU +7.1%) by leveraging accumulated multi-view context.
slammappingembodied perceptionneural descriptor map
OpenReview ↗
PDF ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 Utonia: Toward One Encoder for All Point Clouds
Yujia Zhang, Xiaoyang Wu, Yunhan Yang, Xianzhe Fan, Han Li, Yuechen Zhang, Zehao Huang, Naiyan Wang, Hengshuang Zhao
We dream of a future where point clouds from all domains can come together to shape a single model that benefits them all. Toward this goal, we present Utonia, a first step toward training a single self-supervised point transformer encoder across heterogeneous domains, spanning remote sensing, outdoor LiDAR, indoor RGB-D sequences, object-centric CAD models, and point clouds lifted from RGB-only videos. Despite their distinct sensing geometries, densities, and priors, Utonia learns a consistent representation space that transfers across domains. This unification improves perception capability while revealing intriguing emergent behaviors that arise only when domains are trained jointly. Beyond perception, we observe that Utonia representations can also benefit embodied and multimodal reasoning: conditioning vision-language-action policies on Utonia features improves robotic manipulation, and integrating them into vision-language models yields gains on spatial reasoning. We hope Utonia can serve as a step toward foundation models for sparse 3D data, and support downstream applications in AR/VR, robotics, and autonomous driving.
point-cloudself-supervised-learningfoundation-model3d-representation
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 6 (Hall A) · Thursday, July 9, 10:30-12:15 Video-Based Optimal Transport for Feedback-Efficient Offline Preference-Based Reinforcement Learning
Tung Minh Luu, Hwanhee Kim, Younghwan Lee, Chang D. Yoo
Conveying complex objectives to reinforcement learning (RL) agents often requires meticulous reward engineering. Preference-based RL (PbRL) offers a promising alternative by learning reward functions from human feedback, but its scalability is hindered by high labeling costs. Inspired by advances in Video Foundation Models (ViFMs), we present Video-based Optimal Transport Preference (VOTP), a semi-supervised framework that learns effective reward functions from only a handful of labels. By leveraging optimal transport to align visual trajectories within the rich representation space of ViFMs, VOTP effectively generates high-fidelity pseudo-labels for large amounts of unlabeled data, substantially reducing human supervision. Extensive experiments across locomotion and manipulation benchmarks demonstrate the superiority of VOTP, which outperforms state-of-the-art offline PbRL methods under limited feedback budgets. We also showcase the robustness of VOTP in the presence of visual distractors and validate its utility on real robotic tasks, where it learns meaningful rewards with minimal human input.
preference-based rloffline rlrobotic manipulationvideo foundation models
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 Vision-Language-Action Pretraining from Large-Scale Human Videos
Hao Luo, Yicheng Feng, Wanpeng Zhang, Sipeng Zheng, Ye Wang, Haoqi Yuan, jiazheng liu, Chaoyi Xu, Haiweng Xu, Qin Jin, Zongqing Lu
Existing Vision-Language-Action (VLA) models struggle with complex manipulation tasks requiring high dexterity and generalization, primarily due to their reliance on synthetic data with significant sim-to-real gaps or limited teleoperated demonstrations.
To address this bottleneck, we propose leveraging human hands as a manipulator template, capitalizing on the rich dexterity and scalability present in web data of human manipulation. Our approach introduces physical instruction tuning, a novel training paradigm that combines large-scale VLA pretraining from human videos, perspective spatial alignment for reasoning in a unified physical space, and post-training adaptation in physical environments. Additionally, we introduce a part-level motion tokenization method that achieves millimeter-level reconstruction accuracy to model precise hand trajectories serving as scalable motion primitives. To support our paradigm, we develop a comprehensive data curation pipeline that integrates heterogeneous sources into a large-scale dataset with millions of motion-based instructional instances. Empirically, our model demonstrates superior performance in hand motion generation and instruction following, adhering to favorable scaling laws with respect to model and data sizes. Importantly, we demonstrate promising capabilities to robotic dexterous manipulation, validating the effectiveness of bridging the human-robot embodiment gap. Project page is available at \url{https://research.beingbeyond.com/being-h0}.
vlahuman video pretrainingdexterous manipulationmotion tokenization
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models
Borong Zhang, Jiahao Li, Jiachen Shen, Yuhao Zhang, Yishuai Cai, Yuanpei Chen, Juntao Dai, Jiaming Ji, Yaodong Yang
While Vision-Language-Action models (VLAs) are rapidly advancing toward generalist robot policies, quantitatively characterizing their capability boundaries and failure modes remains challenging. To address this, we introduce **VLA-Arena**, a comprehensive benchmark. It features a novel structured task design framework to quantify difficulty across three orthogonal axes: **(1) Task Structure**, **(2) Language Command**, and **(3) Visual Observation**. This allows us to systematically design tasks with fine-grained difficulty levels, enabling a precise measurement of model capability frontiers. For task structure, VLA-Arena comprises 11 task suites organized into four dimensions: **Safety**, **Distractor**, **Extrapolation**, and **Long Horizon**, totaling 170 tasks. Each suite spans three difficulty levels (L0–L2), with fine-tuning restricted to L0 to rigorously assess generalization. Orthogonal to this, language (W0-W4) and visual (V0-V4) perturbations can be applied to any task as diagnostic probes to distinguish robust grounding from superficial pattern matching. Our extensive evaluation of state-of-the-art VLAs reveals critical limitations: memorization over generalization, superficial visual perception, and a neglect of safety constraints. Additionally, model rank reversals across L0–L2 validate that each level provides non-redundant insights. To foster research addressing these model limitations and ensure reproducibility, we provide the complete VLA-Arena framework, including an end-to-end toolchain from task definition to automated evaluation and the VLA-Arena-S/M/L datasets for fine-tuning. Our benchmark, datasets, models, and leaderboard are publicly available at https://vla-arena.github.io.
vla benchmarkrobot manipulationsafetygeneralization
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 VLA-ATTC: Adaptive Test-Time Compute for VLA Models with Relative Action Critic Model
Wenhao Li, Xiu Su, Yichao Cao, Hongyan Xu, Xiaobo Xia, Shan You, Yi Chen, Chang Xu
Vision-Language-Action (VLA) models have demonstrated remarkable capabilities and generalization in embodied manipulation. However, their decision-making relies on a fast, instinctive process that lacks deliberation. This strategy often leads to suboptimal or catastrophic actions when facing complex or ambiguous scenarios that require greater consideration. In this paper, we introduce \textbf{VLA-ATTC}, a framework that endows VLA models with adaptive test-time compute (TTC). VLA-ATTC employs an uncertainty-based ``cognitive clutch'' to dynamically transition from reflexive execution to a TTC deliberation phase when necessary. During TTC phase, a novel \textbf{Relative Action Critic} (RAC) model identifies the optimal action from generated candidates via pairwise comparisons. This relative mechanism replaces unstable absolute value estimation, significantly simplifying the learning objective. Furthermore, we introduce an efficient sampling strategy to amortize computational costs and an automated data pipeline that curates preference pairs without manual annotation. On the LIBERO-LONG benchmark, VLA-ATTC reduces the failure rate of the SOTA model PI0.5 by over 50\%.
vlatest-time computeaction criticlibero
OpenReview ↗
PDF ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 VLANeXt: Recipes for Building Strong VLA Models
Xiao-Ming Wu, Bin Fan, Kang Liao, Jian-Jian Jiang, Runze Yang, Yihang Luo, Zhonghua Wu, Wei-Shi Zheng, Chen Change Loy
Following the rise of large foundation models, Vision–Language–Action models (VLAs) emerged, leveraging strong visual and language understanding from Vision-Language Models for general-purpose policy learning. Yet, the current VLA landscape remains fragmented and exploratory. Although many groups have proposed their own VLA models, inconsistencies in training protocols and evaluation settings make it difficult to identify which design choices truly matter. To bring structure to this evolving space, we reexamine the VLA design space under a unified framework and evaluation setup. Starting from a simple VLA baseline similar to RT-2, which is the origin of VLA, we systematically dissect design choices along three dimensions: foundational components, perception essentials, and action modelling perspectives. From this study, we distill 12 key findings that together form a practical recipe for building strong VLA models. The outcome of this exploration is a simple yet effective model, VLANeXt. It outperforms the state-of-the-art methods on the LIBERO and LIBERO-plus benchmarks and demonstrates strong performance in real-world experiments. We release a unified and easy-to-use codebase to reproduce our findings, explore the design space, and develop new VLA variants on top of a shared foundation. The codebase is available at https://github.com/DravenALG/VLANeXt.
vlarobot learningdesign spacereal-world
OpenReview ↗
PDF ↗
Code ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Model
Yanjiang Guo, Tony Lee, Lucy Xiaoyang Shi, Jianyu Chen, Percy Liang, Chelsea Finn
The goal of this paper is to improve the performance and reliability of vision-language-action (VLA) models through iterative online interaction.
Since collecting policy rollouts in the real world is expensive, we investigate whether a learned simulator—specifically, an action-conditioned video generation model—can be used to generate additional rollout data.
Unfortunately, existing world models lack the physical fidelity necessary for policy improvement: they are predominantly trained on demonstration datasets that lack coverage of many different physical interactions (particularly failure cases) and struggle to accurately model small yet critical physical details in contact-rich object manipulation.
We propose a simple iterative improvement algorithm that uses real-world roll-out data to improve the fidelity of the world model, which can then, in turn, be used to generate supplemental synthetic data for improving the VLA model.
In our experiments on a real robot, we use this approach to improve the performance of a state-of-the-art VLA model on multiple downstream tasks.
We achieve a 39.2\% absolute success rate improvement over the base policy and 11.6\% improvement from training with the generated synthetic rollouts. Videos can be found at this anonymous website: \url{https://sites.google.com/view/vla-w}.
vlaworld modelreal robotiterative training
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 3 (Hall A) · Wednesday, July 8, 10:30-12:15 VSCD: Video-based Scene Change Detection in Unaligned Scenes
Jiae Yoon, Ue-Hwan Kim
Detecting what has changed in an environment is essential for long-term autonomy, yet most change detection settings assume fixed viewpoints, mild misalignment, or only a few changed objects. We introduce Video-based Scene Change Detection (VSCD), which predicts a pixel-wise change mask for each query frame, given a reference and a query RGB video of the same indoor space recorded at different times under unconstrained camera motion. The two videos are not temporally synchronized, and many object instances may appear or disappear. To study this setting, we build a large-scale benchmark with over 1.1 million frames annotated with pixel-accurate change masks, together with a real-world test set for evaluating transfer beyond simulation. We propose a query-centric multi-reference model that learns temporal matching implicitly from change-mask supervision, aligns candidate reference features to the query via local patch correspondence, and fuses per-candidate change features using frame-level and patch-level confidence before decoding a high-resolution mask once per frame. Our approach achieves state-of-the-art performance against strong image- and video-based baselines, and we validate its real-world impact by deploying it on a mobile robot for two downstream applications—visual surveillance and object incremental learning.
scene change detectionmobile robotlong-term autonomybenchmark
OpenReview ↗
PDF ↗
Code ↗
Poster Session 4 (Hall A) · Wednesday, July 8, 2:30-4:15 WestWorld: A Knowledge-Encoded Scalable Trajectory World Model for Diverse Robotic Systems
Yuchen Wang, Jiangtao Kong, Sizhe Wei, Xiaochang Li, Haohong Lin, Hongjue Zhao, Tianyi Zhou, Lu Gan, Huajie Shao
Trajectory world models play a crucial role in robotic dynamics learning, planning, and control. While recent works have explored trajectory world models for diverse robotic systems, they struggle to scale to a large number of distinct system dynamics and overlook domain knowledge of physical structures. To address these limitations, we introduce *WestWorld*, a kno**W**ledge-**E**ncoded **S**calable **T**rajectory **World** model for diverse robotic systems. To tackle the scalability challenge, we propose a novel system-aware Mixture-of-Experts (Sys-MoE) that dynamically combines and routes specialized experts for different robotic systems via a learnable system embedding. To further enhance zero-shot generalization, we incorporate domain knowledge of robot physical structures by introducing a structural embedding that aligns trajectory representations with morphological information. After pretraining on 89 complex environments spanning diverse morphologies across both simulation and real-world settings, *WestWorld* achieves significant improvements over competitive baselines in zero- and few-shot trajectory prediction. Additionally, it shows strong scalability across a wide range of robotic environments and significantly improves performance on downstream model-based control for different robots. Finally, we deploy our model on a real-world Unitree Go1, where it demonstrates stable locomotion performance. The code is available at https://github.com/511205787/WestWorld.
world modelrobot learningtrajectory predictionquadrupedmixture-of-experts
OpenReview ↗
PDF ↗
Code ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 What You Think is What You See: Driving Exploration in VLM Agents via Visual-Linguistic Curiosity
Haoxi Li, Qinglin Hou, Jianfei Ma, Jinxiang Lai, Tao Han, Sikai Bai, Jingcai Guo, Jie ZHANG, Song Guo
To navigate partially observable visual environments, recent VLM agents increasingly internalize world modeling capabilities directly into their policies via explicit CoT reasoning with reinforcement learning (RL). However, mere passive exploitation of reasoning on visited states is insufficient for sparse-reward agentic tasks, as it lacks the epistemic drive to actively uncover the *known unknown* required for robust generalization. We ask: *Can VLM agents actively find signals that challenge and update their internal world model through curiosity-driven exploration?* In this work, we propose **GLANCE**, a unified framework that bridges reasoning and exploration by grounding the agent's linguistic world model into the stable visual representations of an evolving target network. Crucially, **GLANCE** leverages the discrepancy between linguistic prediction and visual reality as an intrinsic curiosity signal within reinforcement learning, steering the agent to actively explore areas where its internal model is uncertain. Extensive experiments across a series of agentic tasks show the effectiveness of **GLANCE**, and demonstrate that aligning *what the agent thinks* with *what the agent sees* is key to solving complex or sparse agentic tasks.
vlm-agentscuriosity-drivenreinforcement-learningexploration
OpenReview ↗
PDF ↗
Poster Session 2 (Hall A) · Tuesday, July 7, 2:00-3:45 When Replanning Becomes the Bottleneck: Budgeted Replanning for Embodied Agents
Shuaijun Liu, Feiyang You, Xingwei Chen, Ningxin Su
Embodied agents replan frequently to recover from execution drift, partial observability, and coordination hazards, but each LLM-based replanning call can consume an accumulated textual context that grows over time and across agents. Once this context becomes large, replanning latency develops heavy tails and can miss real-time deadlines even when task success remains high, a failure mode that is hard to detect from average latency or success alone. We present BRACE, a controller that formulates replanning as a budgeted control loop by deciding whether to replan, selecting a replanning mode, and allocating an explicit token budget and latency service-level objective (SLO) while accounting for optional efficiency modules. As a reusable component, we introduce E-RECAP, a cost-aware progressive token pruning method that predicts token utility and prunes replanning contexts across transformer layers while preserving critical head and tail tokens. Across Meta Habitat, RoboFactory, and AirSim, BRACE with E-RECAP reduces replanning-call token counts by 62–92% and SLO violation rates from 85.5–100.0% to 4.7–50.0% in settings where task success is already saturated. In a harder RoboFactory setting where open-loop, frozen-plan, and No BRACE all fail, BRACE + E-RECAP reaches 80.0% success with 4.6% SLO violations, demonstrating that tail-aware per-call budgeting is effective across embodied platforms.
embodied agentsreplanningllm planninglatencybenchmark
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 World Guidance: World Modeling in Condition Space for Action Generation
Yue Su, Sijin Chen, Haixin Shi, Mingyu Liu, Zhengshen Zhang, Ningyuan Huang, Weiheng Zhong, Zhengbang Zhu, Yuxiao Liu, Xihui Liu
Leveraging future observation modeling to facilitate action generation presents a promising avenue for enhancing the capabilities of Vision-Language-Action (VLA) models.
However, existing approaches struggle to strike a balance between maintaining efficient, predictable future representations and preserving sufficient fine-grained information to guide precise action generation.
To address this limitation, we propose WoG (World Guidance), a framework that maps future observations into compact conditions by injecting them into the action inference pipeline.
The VLA is then trained to simultaneously predict these compressed conditions alongside future actions, thereby achieving effective world modeling within the condition space for action inference.
We demonstrate that modeling and predicting this condition space not only facilitates fine-grained action generation but also exhibits superior generalization capabilities. Moreover, it learns effectively from substantial human manipulation videos.
Extensive experiments across both simulation and real-world environments validate that WoG significantly outperforms existing methods based on future prediction. Project page is available at: https://selen-suyue.github.io/WoGNet/.
vlaworld modelaction generationimitation learning
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 XR-1: Towards Versatile Vision-Language-Action Models via Learning Unified Vision-Motion Representations
Shichao Fan, Kun Wu, Zhengping Che, Xinhua Wang, Di Wu, Fei Liao, Ning Liu, Yixue Zhang, Zhen Zhao, Zhiyuan Xu, Meng Li, Qingjie Liu, Shanghang Zhang, Min Wan, Jian Tang
Recent progress in large-scale robotic datasets and vision-language models (VLMs) has advanced research on vision-language-action (VLA) models. However, existing VLA models still face two fundamental challenges: (\textit{i}) producing precise low-level actions from high-dimensional observations, (\textit{ii}) bridging domain gaps across heterogeneous data sources, including diverse robot embodiments and human demonstrations. Existing methods often encode latent variables from either visual dynamics or robotic actions to guide policy learning, but they fail to fully exploit the complementary multi-modal knowledge present in large-scale, heterogeneous datasets. In this work, we present \textbf{XR-1}, a novel framework for versatile and scalable VLA learning across diverse robots, tasks, and environments.
At its core, XR-1 introduces the \emph{Unified Vision-Motion Codes (UVMC)}, a discrete latent representation learned via a dual-branch VQ-VAE that jointly encodes visual dynamics and robotic motion. UVMC addresses these challenges by (\textit{i}) serving as an intermediate representation between the observations and actions, and (\textit{ii}) aligning multimodal dynamic information from heterogeneous data sources to capture complementary knowledge. To effectively exploit UVMC, we propose a \emph{three-stage training paradigm}: (\textit{i}) self-supervised UVMC learning, (\textit{ii}) UVMC-guided pretraining on large-scale cross-embodiment robotic datasets, and (\textit{iii}) task-specific post-training. We validate XR-1 through extensive real-world experiments with more than 12,000 rollouts on six different robot embodiments, spanning over 120 diverse manipulation tasks. XR-1 consistently outperforms state-of-the-art baselines such as $\pi_0$ and GR00T-N1.5 while demonstrating strong generalization to novel objects, background variations, distractors, and illumination changes. Our project is at \href{https://xr-1-vla.github.io/}{https://xr-1-vla.github.io/}.
vlacross-embodimentimitation learningunified representation
OpenReview ↗
PDF ↗
Project page ↗
Poster Session 5 (Hall A) · Wednesday, July 8, 5:00-6:45 Zero-shot Active Mapping via Fused 360-BEV Representations and Vision–Language Models
Yuanze Wang, Dianxi Shi, Yuetian Wang, Shiming Song, Haikuo Peng, Chunping Qiu, Mengzhu Wang
Active mapping enables embodied agents to understand and interact in previously unseen environments. However, most methods struggle to achieve zero-shot generalization to large-scale scenes and lack support for language instructions. We propose a VLM-based active mapping method that achieves zero-shot mapping while facilitating language-driven human–agent interaction. First, we introduce a 360-BEV representation that integrates omnidirectional semantics with BEV-aligned geometric structure to enhance scene understanding. Second, we develop a candidate waypoint generation strategy that allows the VLM-driven agent to select informative 2D waypoints in image space and back-project them into executable metric actions in 3D space, enabling the VLM to plan in its strongest modality. Third, we design a VLM-based depth-first exploration agent that decomposes the scenes into explorable regions, selects informative waypoints within each region, and organizes them into a topological tree. The agent follows the depth-first exploration policy to achieve thorough coverage of large-scale scenes. Without task-specific training, our method outperforms the strongest baseline, improving coverage and AUC by approximately 13.25\% and 14.00\%, respectively, while enabling language-conditioned interaction.
active mappingvision-language-modelsembodied agentsexplorationbev-representation
OpenReview ↗
PDF ↗
Poster Session 7 (Hall A) · Thursday, July 9, 2:30-4:15 