Papers for 2026-11-08

10 papers

Latent RL Priors for Data-Efficient Dexterous Manipulation

3.0/5 一般 裁判分 4.0 Candidate 当日相对
Yixiao Wang, Peiyi Chen, Mustafa Mukadam, ... , Masayoshi Tomizuka, Son Tran
总结: 用仿真控制先验,少演示学习视觉灵巧操作。
方法: 蒸馏强化学习为流匹配解码器,冻结后只靠动作监督学视觉接口。
证据: 三个仿真任务上优于全参数微调视觉策略。
为什么适合我: 属灵巧手操作,与人形足式全身控制关联有限。
推荐理由: 最接近人形移动操作与深度强化学习:用仿真RL先验做灵巧操作的视觉策略适配,但是手部/双臂操作而非全身运动跟踪或足式运动。
原摘要

Learning dexterous manipulation from limited visual demonstrations is challenging because policies must acquire both perceptual grounding and precise control, while reinforcement learning (RL) in simulation can acquire rich control behaviors from large-scale interaction with privileged information. We introduce \textbf{Latent RL Prior}, a two-stage framework that reuses this control knowledge for data-efficient visual policy learning. During privileged pretraining, we distill large-scale RL rollouts into a flow-matching action decoder conditioned on proprioception and latent task context. During visual adaptation, we freeze the decoder and learn a visual interface that maps images to its latent context using action supervision alone, without privileged labels or latent targets. The same decoder can further be shared across arms by learning coordinated visual contexts for bimanual manipulation. Across three simulated tasks, our method outperforms full-parameter fine-tuning of $π_{0.5}$ and GR00T N1.7 in the low-data regime; on door lifting and bimanual assembly, it surpasses full-data baselines using $10\times$ and $5\times$ fewer demonstrations, respectively. Further analysis shows that direct visual-to-latent conditioning improves adaptation performance and training stability over routing visual predictions through the pretrained state encoder. Robustness to unseen mass variation and greater linear accessibility of contact-force information in decoder representations further provide evidence of retained RL-acquired control knowledge. Project website: https://yixiaowang7.github.io/latent-rl-prior-page/.

Teaching Agents the "Principle", Not the "Reward"

3.0/5 一般 裁判分 1.0 Candidate 当日相对
Shan Yang
总结: 直接教授行为原则,减轻从奖励反推意图的负担。
方法: 用关键量与参考逐步指示平衡、蓄能与平稳落地。
证据: 实验显示智能体能学到较好且可泛化的策略。
为什么适合我: 平衡与落地原则或可启发足式高动态控制。
推荐理由: 以平衡、蓄能与平稳落地等行为原则替代奖励,方向上接近物理角色控制与敏捷运动,但未明确做人形全身跟踪、重定向或仿真到现实。
原摘要

Reinforcement learning has achieved strong performance in many tasks by letting agents learn from nothing more than a reward signal. However, in more complex tasks, agents still struggle to learn good, generalizable policies, even with carefully designed rewards and abundant samples. We argue that this difficulty stems from the reward decoding burden: designers encode their knowledge of the desired behavior into rewards and penalties, and agents must infer this knowledge from long sequences of noisy scalars. To address this burden, we propose prescriptive policy learning (PPL), which directly teaches agents higher-level knowledge about the tasks, termed principles of behavior, such as how to maintain balance, how to accumulate energy, and how to land smoothly. Through appropriately selected key quantities and references, PPL can directly indicate at every step whether the agent follows the principles and to what extent. Our experiments show that agents can learn good, generalizable policies without any reward. Taught only simple heuristic principles, PPL agents outperform reward-based methods with far fewer samples. Their policies also transfer directly to harder settings and keep performing well, demonstrating their strong generalization ability. In summary, PPL provides a new approach in which rough knowledge of the desired behavior is enough for agents to learn good, generalizable policies, even when the system dynamics cannot be accurately modeled.

Compounding Physical RSI: Turning Execution Failure into Inherited Capability

2.0/5 偏低 裁判分 5.0 Candidate 当日相对
Yixuan Yang, Qiang Chen, Run Shao, ... , Yichao Cao, Xiu Su
总结: 把执行失败变成可继承能力,推动物理策略递归改进。
方法: 针对阶段难定位、失败无纠正和技能遗忘设计自我进化。
证据: 任务完成近似各阶段通过率之积,收益可跨代放大。
为什么适合我: 面向通用操作进化,不是运动跟踪或跑酷。
推荐理由: 讨论通用操作策略的分阶段失败诊断与递归自我改进,仅弱相关于机器人学习;未涉及人形/足式全身控制、运动跟踪或重定向。
原摘要

Generalist robot policies are trained on static offline datasets, yet physical manipulation unfolds sequentially across stages. Because a weakness in a downstream stage remains unobservable until the policy passes earlier ones, each resolved bottleneck exposes the next hidden failure mode. This staged structure calls for Recursive Self-Improvement (RSI), in which a policy iteratively executes, diagnoses failures, and updates itself. Moreover, because task completion behaves like a product of stage-level pass rates, recursive gains can multiply across generations. In physical environments, three bottlenecks obstruct compounding: terminal execution feedback rarely localizes the stage that caused a failure, failed trajectories do not supply the corrective action, and updating shared parameters can degrade previously mastered skills. We propose Compounding Physical RSI, a self-evolving policy framework that turns execution failures into heritable capability. First, automated capability-gap assessment traces failed rollouts to the task stage where progression breaks down and formulates a structured improvement objective. Second, conditioned on this objective, our remedy stage curates validated, stage-relevant experience and refines the policy via behavior-regularized offline reinforcement learning, without requiring costly online rollouts at each optimization step. Third, to safeguard existing skills, our retention stage constructs weight-space candidates between the inherited reference policy and the refined policy, evaluates them across the task suite, and promotes the best candidate as the reference state for the next generation. Evaluated across all 42 tasks of the RoboDojo-Sim benchmark, CSU-AI-07 ranks first on the leaderboard with a three-seed mean Overall Score of 26.15 and Success Rate of 20.70%, outperforming the next-best entry by 5.92 points and 5.82 percentage points.

ODE Policy: Robot Learning via Action Dynamics, Not Actions

2.0/5 偏低 裁判分 4.0 Candidate 当日相对
Jindou Jia, Jingliang Li, Kuangji Zuo, ... , Junpeng Zhang, Jianfei Yang
总结: 学习动作向量场而非离散动作,使轨迹天然连续。
方法: 神经微分方程匹配模仿动作块或强化学习单步,并用多样性头。
证据: 覆盖单臂、双臂、灵巧、运动和移动操作,对比十四个基线。
为什么适合我: 直接涉及运动与移动操作,贴合全身跟踪兴趣。
推荐理由: 提出学习动作动态的通用机器人策略,不是针对人形全身控制或运动跟踪。
原摘要

Most robot policies, whether trained by imitation learning (IL) or reinforcement learning (RL), map observations to action values at discrete control steps and leave the continuity of physical motion to be learned implicitly. We propose ODE policy, which learns the dynamics of the action rather than the action itself. A neural ordinary differential equation (ODE) learns the vector field of the action by matching the integrated actions to the targets over short action chunks in IL and single control steps in RL. A lightweight diversity head captures multi-modal behavior by randomizing the direction of the field. At deployment, each action is integrated from the previous one, so the executed trajectory is continuous by construction. We evaluate it as a visuomotor policy, as the action head of vision-language-action (VLA) and world action models (WAMs), and as the actor of model-based RL against 14 baselines on unimanual, bimanual, dexterous, locomotion, and loco-manipulation tasks. In IL, against flow matching, it reaches 5.9$\times$ the success, 5.4$\times$ the generalization, and 3.9$\times$ the mode balance after one epoch, and generates actions 10.8$\times$ faster. It mostly moves more smoothly than human demonstrations. On 14 real-world tasks across single-arm, dual-arm, dexterous, and humanoid robots, it achieves the highest success rate. In RL, it further speeds up SuperMimic for humanoid motion tracking and smooths its actions. Efficient in learning, data, parameters, and execution, ODE policy offers a promising path to scaling robot learning.

Yuanchu Liang, Yue Yang, An-Chi He, Hanna Kurniawati
总结: 用潜噪声共享表示,高效且有保证地微调扩散策略。
方法: 推导桥方程消除噪声混叠,对齐噪声空间与动作空间价值。
证据: 该微调框架落入既有理论并具近最优保证。
为什么适合我: 可加速扩散策略强化学习,但未针对足式全身。
推荐理由: 通用扩散策略的强化学习微调理论,并非人形运动生成或全身控制。
原摘要

Pre-trained diffusion and flow policies have emerged as powerful backbones for robotics, yet adapting them to novel data via reinforcement learning (RL) remains computationally expensive, sample inefficient and lacks theoretical guarantees. Prior approaches that fine-tune the initial noise bypass costly backpropagation through time but struggle with the "noise aliasing" problem: there does not exista natural link between the noise space value and the action space value. This paper resolves these challenges by introducing the Latent Noise Linear Markov Decision Process (LNL-MDP). We derive the Bridge Equation for LNL-MDPs which eliminates the noise aliasing problem by aligning values with a shared representation between the noise and the action space. Furthermore, our LNL-MDP fine tuning framework falls under the FineTuneRL theory with near-optimal guarantee. Leveraging these insights, we propose Latent Representation Bridging for Diffusion (LARBRID), an algorithm that efficiently learns the representation to optimize the initial distribution that steers the pre-trained model to act optimally in the environment. It yields up to 2-3$\times$ performance increases on challenging control problems across a diverse suite of robotic locomotion and manipulation tasks. Code can be found at: github.com/Maincode-Research/LARBRID.git.

Yijiong Lin, Wenhao He, Toby Jack, ... , Haoran Li, Nathan F. Lepora
总结: 用高效剪切触觉仿真支持接触丰富的仿真到现实操作。
方法: 刚体仿真估计接触切向光流,结合深度场与本体感觉训练策略。
证据: 建于IsaacLab,避免高成本软体仿真以扩展强化学习。
为什么适合我: 仿真到现实和接触感知可借鉴,对象是灵巧手。
推荐理由: 灵巧手触觉仿真与仿真到现实,虽涉及仿真迁移,但是触觉操作而非人形全身运动。
原摘要

Robotic simulators that represent shear within tactile sensing are becoming increasingly important for contact-rich manipulation. However, simulating contact shear with high-fidelity deformable or soft-body simulation is computationally demanding, limiting scalability to large-scale reinforcement learning. Here we introduce \textit{TactileLab}, a scalable multimodal tactile simulation framework built on IsaacLab, with a particular focus on a new optical-flow-based tactile representation termed \textit{simulated tactile flow}. Instead of relying on soft-body physics to model shear deformation, TactileLab uses rigid-body simulation to efficiently estimate dense tangential motion at the contact interface, producing tactile flow fields that encode shear-relevant contact dynamics. This flow field complements geometric features from a contact depth field. We further introduce a novel policy architecture for jointly processing long-horizon proprioceptive signals and rich tactile observations. Across multiple contact-rich tasks, policies trained using simulated tactile flow learn efficiently at scale and exhibit emergent contact-aware behaviours. To facilitate sim-to-real policy deployment, we then propose a novel real-to-sim tactile translation method that maps real tactile image streams to simulated tactile flow and depth fields, together with a new tactile data collection method, and validate it across diverse tactile sensor designs. Using this pipeline, we demonstrate real-world deployment on tasks encompassing dual-arm and dexterous in-hand manipulation. Overall, TactileLab provides an end-to-end sim-to-real framework for scalable tactile policy learning, enabling diverse shear-sensitive manipulation across heterogeneous tactile robotic systems.

Beyond Terminal Constraints: Reliable Robot Control with Learned Safe Sets

2.0/5 偏低 裁判分 3.0 Candidate 当日相对
Gianni Lunardi, Elias Fontanari, Matteo Saveriano, Andrea Del Prete
总结: 放宽控制不变集,用更广安全集保证预测控制安全。
方法: 约束预测时域多个节点,并以安全中止弥补递归可行性缺失。
证据: 六关节机械臂仿真显示约束违反更少。
为什么适合我: 安全模型预测控制可借鉴,但验证对象不是人形。
推荐理由: 研究MPC与强化学习的安全集理论,属于通用机器人安全控制,不是人形全身MPC或运动跟踪。
原摘要

Ensuring safety in advanced robot control methods such as Model Predictive Control (MPC) and Reinforcement Learning remains an open challenge. A common approach relies on control-invariant sets, yet their exact computation for nonlinear systems is generally intractable, and numerical approximations compromise theoretical guarantees. In this work, we show that safety does not need to rely exclusively on control invariance. We introduce novel classes of safe sets, each broader than classical control-invariant sets, and analyze corresponding MPC schemes that guarantee recursive feasibility by constraining different nodes of the prediction horizon rather than only the terminal state. Furthermore, when the safe set is an inner approximation of a suitable control-invariant set, our methods can ensure safety through a safe-abort strategy, even in the absence of recursive feasibility. We validate our approaches in extensive simulations on a 6-joint robot manipulator, showing fewer constraint violations than state-of-the-art methods while maintaining comparable tracking performance and computational efficiency. Hardware experiments further demonstrate the practicality of our schemes for real-world robotic applications.

Diversity-Aware Experience Replay via Statistical Leverage Scores

2.0/5 偏低 裁判分 3.0 Candidate 当日相对
Shaifalee Saxena, Alexander Scheinker
总结: 按数据罕见程度重采样经验,突出物理机器人的边缘事件。
方法: 用高峰度特征计算统计杠杆分,再与均匀采样混合组批。
证据: 分数只来自缓存数值,不依赖标签、奖励或价值网络。
为什么适合我: 或利于保留稀有失稳样本,但是通用回放方法。
推荐理由: 通用离策略经验回放算法,虽提及物理机器人数据分布,但不针对人形全身控制、运动模仿或感知运动。
原摘要

Off-policy deep reinforcement learning trains on random samples from a replay buffer of past experience. In physical robot settings, that buffer is dominated by routine: thousands of near-identical transitions from normal operation, while the rare events that define the edges of the task are rarely sampled. We propose scoring every stored transition by its statistical leverage, which measures how unusual it is relative to the buffer's own data, computed from the stored numbers alone, with no labels, rewards, or value network. We draw each mini-batch from a mixture of this score and uniform sampling. First, the natural way to build the score along the directions where the data varies most turns out to be blind when actions are stored as one-hot indicators, because indicator columns always win the variance ranking; we instead select the feature columns with the heaviest tails (largest kurtosis) where small far-away populations are revealed. Second, the importance weights that prioritized replay uses to keep gradients unbiased cancel the method exactly: with full correction, training matches uniform sampling, so we drop the correction and deliberately optimize the rebalanced objective. Third, when the rare situations of a log are relational, visible only in how a stretch of behavior relates to its goal, not in any single stored coordinate-per-transition scores are blind; applying the same tail rule to short trajectory windows described relative to their goal restores the signal, and on a three-joint arm that must push an object around an obstacle it raises the rare-task success from 0.49 to 0.69 against a labeled-oracle ceiling of 0.74.

Roko Parac, Christos Ziakas, Leo Ardon, Alessandra Russo
总结: 语言生成奖励机,视觉模型把命题落到图像并给稠密奖励。
方法: 大语言模型构建自动机,视觉语言模型估计各转移进度。
证据: 用于图像观测下的奖励机引导策略学习与评估。
为什么适合我: 侧重视觉任务奖励结构,与全身运动跟踪关联弱。
推荐理由: 用语言模型与视觉语言模型构造奖励机,属于通用任务型强化学习,未落到人形移动操作或感知足式运动。
原摘要

Reward machines (RMs) have been proposed as a reward formulation in reinforcement learning (RL) to capture non-Markovian tasks where credit assignment depends on history. By representing reward functions as automata, RMs expose the task's temporal structure and allow such rewards to be handled within a Markovian framework. However, they assume a labelling function, a proposition-grounding oracle for observations, which must be manually engineered for each environment, limiting their wider applicability. In this work, we address this limitation by proposing RM-VLM, a novel approach in which a large language model (LLM) generates an RM from a natural-language task description, and a vision-language model (VLM) acts as the labelling function, grounding the RM's propositions over image-based observations during RM-guided policy learning. A VLM-based value function estimator is used to assess the agent's progress towards each RM transition and derive a dense reward signal. We evaluate RM-VLM on three domains covering non-Markovian grid-world navigation, robotic manipulation, and 3D object collection. Our experimental results show that RM-VLM consistently achieves higher task success rates than existing VLM-based reward methods across all tested domains.

Shu'ang Sun, Yibo Qiu, Haoliang Ye, ... , Ronald Xu, Mingzhai Sun
总结: 数字孪生为真实实验室操作提供初始策略和稠密奖励。
方法: 高斯泼溅加物理仿真产奖励,蒸馏后结合执行历史细化策略。
证据: 五项真实任务最终平均自主成功率达百分之九十八。
为什么适合我: 仿真到现实奖励迁移可参考,任务不是足式机器人。
推荐理由: 数字孪生服务于实验室操作的在线强化学习,不属于人形或足式全身控制与移动操作。
原摘要

Online reinforcement learning offers an effective approach to precise laboratory manipulation, yet costly real-world demonstrations and limited access to dense process rewards hinder its practical application. We present TwinBridge, a digital twin-driven framework that supplies policy initialization and dense visual rewards for real-world online learning. We construct digital twins by combining 3D Gaussian splatting with physics simulation. Here, simulator states provide task events and geometric progress under physical prerequisites and ordering constraints, yielding rewards for policy pretraining with simulated demonstrations and reinforcement learning. We distill this supervision into a visual reward model. During real-world interaction, its predictions are combined with execution history to compute dense rewards for human-in-the-loop policy refinement. Across five real-world laboratory tasks, TwinBridge achieves a final mean autonomous success rate of 98% with at most 25 minutes of real-world interaction per task. Experiments show that dense rewards improve interaction efficiency over sparse rewards, and simulation-based policy initialization further accelerates learning.