Papers for 2026-10-27

10 papers

ViT-H: Generative Video-To-Humanoid Control with Motion Tracking Feedback

5.0/5 很相关 裁判分 10.0 Top pick 当日相对
Zimo He, Tianyi Bian, Zizhong Yang, ... , Siyuan Huang, Yixin Zhu
总结: 从单目视频直接生成人形参考运动并用跟踪反馈优化。
方法: 扩散模型生成运动,跟踪结果DPO后训练,自回归支持实时遥操作。
证据: Unitree G1上运动质量与可执行性优于重建重定向跟踪流水线。
为什么适合我: 视频生成加跟踪反馈,契合人体动作可跟踪可上真机需求。
推荐理由: 直接对应人形全身控制与运动跟踪:单目视频经扩散生成可执行参考动作,再用跟踪反馈做偏好优化,并含G1实时遥操作。
原摘要

Learning humanoid control from monocular human videos offers a scalable way to acquire diverse whole-body behaviors. Yet, existing reconstruction-retargeting-tracking pipelines are limited in system-level scalability by separately optimized intermediate stages and cannot directly optimize reference motion executability using downstream tracking feedback. We present ViT-H, a video-to-humanoid control framework following a generation-and-tracking paradigm. ViT-H directly generates humanoid reference motions from videos with a diffusion model, bypassing human motion reconstruction and retargeting, and uses motion-tracking outcomes to post-train the generator with direct preference optimization. We further introduce an autoregressive variant that enables real-time humanoid video teleoperation with sub-400 ms latency. Extensive experiments on the Unitree G1 demonstrate that ViT-H outperforms state-of-the-art reconstruction-retargeting-tracking pipelines in both motion quality and executability. We further show that ViT-H scales favorably with model size and with both pre-training and post-training data scale, while motion-tracking feedback consistently improves executability across different motion trackers.

Ziyu Lin, Shihang Du, Tianhang Wang, ... , Alois Knoll, Guang Chen
总结: 双人形协作搬运需一致跨智能体与物体接触支撑关系。
方法: 接触支撑结构化重定向,联合优化两机器人围绕物体轨迹。
证据: 新数据集含多任务地形物体序列,保留抓取与支撑结构。
为什么适合我: 接触丰富双人形协作,匹配loco-manipulation与全身接触兴趣。
推荐理由: 紧贴人体动作重定向与接触丰富全身交互:双人形协同搬运,围绕共享物体与支撑关系做接触-支撑结构化重定向。
原摘要

Recent advances in large-scale human motion data, motion retargeting, and motion imitation have enabled increasingly capable whole-body control for individual humanoids. Cooperative carrying is more challenging because successful coordination requires not only plausible individual motions, but also consistent cross-agent and agent-object relations, including shared grasp, relative configuration, and whole-body support. We introduce CoCarry, a cooperative-carrying dataset containing 220 synchronized two-person sequences across 21 task schemes, five shared-object geometries, four terrain types, and approximately 61 minutes of motion. We further propose Contact-Support-Structured Retargeting, which starts from nominal individual motions and jointly refines both humanoids around the same prescribed object trajectory. Cooperation-aware object-local hand anchors preserve compatible four-hand grasp relations, while complementary upper object-interaction and lower support structures retain object-relative geometry and coordinated support. Across all 220 CoCarry sequences retargeted to two A3 humanoids, our method achieves 0.9995 contact preservation, reduces penetration duration by 67.0% and 71.0% relative to GMR and OmniRetarget, respectively, and maintains a 100% retargeting success rate. On CORE4D with Unitree G1 humanoids, it achieves 0.9906 contact preservation with zero measured sliding. We further validate the generated references through physics simulation and multiple real-world cooperative carrying experiments on two physical A3 humanoids, demonstrating their usability for downstream dual-humanoid control.

Robot Motion Transition with a Human Motion Prior

5.0/5 很相关 裁判分 9.0 Top pick 当日相对
Taehyun Byun, Taemoon Jeong, Yoonbyung Chai, ... , Kyungjae Lee, Sungjoon Choi
总结: 用人体运动先验连接独立获取的人形运动成连续序列。
方法: 机器人到人到机器人过程,对齐模型确定过渡边界条件。
证据: 框架利用丰富先验生成合适过渡,避免不当插值动作。
为什么适合我: 运动过渡与重定向,支持人体动作可跟踪可上真机。
推荐理由: 核心就是人形运动跟踪与人体运动先验:用人体空间做时空对齐和动作过渡,把独立片段连成可跨本体复用的连续全身动作。
原摘要

Humanoid motion-tracking policies can reproduce diverse behaviors, yet connecting independently acquired motions into a continuous sequence remains challenging. Interpolation connects boundary poses without considering whether their relative position, heading, and timing provide suitable conditions for a transition. A generative motion prior can produce richer motion, but poorly arranged boundaries may cause it to generate excessive locomotion, turning, or another unintended action. We present Spatio-Temporal Alignment with a Motion Prior (STAMP), a framework for connecting independently acquired humanoid motions through a Robot-to-Human-to-Robot process. STAMP uses human motion space to draw on rich motion priors and reuse transition knowledge across robot embodiments. Its key component is a learned alignment model that determines the boundary conditions for motion inbetweening. Given the two motions retargeted to a human rig, the alignment model ranks candidate positions, headings, and transition durations for the following motion. A human motion prior generates the transition under the selected alignment. The completed sequence is retargeted to the robot and used as a reference to train a motion-tracking policy. We evaluate the tracking success of the generated transitions on Unitree~G1 and~ROBOTIS K1. Physical robot demonstrations further show that STAMP connects independently acquired motions into long-horizon sequences. Project page: https://childtoy.github.io/STAMP-project

Xihan Ma, Bo Zhang, Boxiong Li, ... , Ziyue Liu, Yucheng Wang
总结: 基于预训练全身控制器的简单配方快速学人形拳击。
方法: 规划器请求身体点目标,冻结控制器执行,解码器映射动作。
证据: 从可用控制器起约七分钟单GPU完成任务学习。
为什么适合我: 复用全身技能高效学交互,利于复杂接触场景控制。
推荐理由: 落在人形全身控制上:冻结全身控制器,上层规划器用模仿初始化再强化学习出拳击交互,属于技能复用而非新的运动跟踪本体。
原摘要

HoloBoxing presents a simple recipe for learning humanoid boxing by building on a pretrained whole-body controller. The recipe separates reusable motor skills from task decisions: a compact planner requests future body-point targets, while the frozen controller handles joint-level execution. A learned decoder maps 32-dimensional actions to structured target windows, and measured-state imitation initializes the planner before reinforcement learning. Task learning then refines interaction behavior while reusing the decoder and controller, avoiding the need to relearn whole-body control. This combination keeps task optimization focused on a compact upper policy and supplies an initial behavior before reward-driven exploration. Starting from that available controller, the recipe accounts for about seven minutes of additional training, including reused representation fitting and a single-GPU task run. This excludes controller pretraining, data preparation, evaluation, and search. Simulated boxing experiments and component controls evaluate the resulting behavior and the roles of imitation initialization and the target representation.

Shaoqin Li, Yi Sun
总结: 训练时延迟随机化拓宽人形运动跟踪器的延迟裕度。
方法: 训练中随机化命令延迟以提升对延迟的鲁棒性。
证据: 40ms延迟存活率平均从15%升至95%,裕度增至50-70ms。
为什么适合我: 增强运动跟踪真机鲁棒性,服务可部署全身控制。
推荐理由: 研究对象就是由单目视频训练的人形运动跟踪策略,贡献集中在指令延迟随机化与sim检验,直接服务真机全身跟踪部署。
原摘要

Learned humanoid controllers are usually trained in one simulator and then checked in a second one before they go near hardware. We ask what that check says about command delay. We train motion-tracking policies for a 24-joint humanoid on five motions taken from monocular video and test them in an independent CPU MuJoCo harness, which we first verify against the training model. Every policy completes its motion in at least 26 of 30 episodes under the training randomization. With 40 ms of constant command delay, however, survival drops to 15% on average (0% - 45%), and the delay margin is only 20 - 30 ms. Randomizing the delay during training (0 - 50 ms) lifts survival at 40 ms to 95% on average (75% - 100%) and moves the margin to 50 - 70 ms. A control arm trained for the same number of extra iterations without delay does not improve, so the gain comes from the randomization itself. This holds for all five motions with three training seeds each, and a policy trained in PhysX behaves the same way. The price is slightly looser tracking (0.02 - 0.06 rad on four motions) and, for the most dynamic motion, less robustness to other perturbations. We also show that the first sim-to-sim gap we saw came from the harness, a missing floor and a wrong timestep, and not from the physics engines. The 0 - 20 ms delay ranges common in locomotion training lie inside the margin an unrandomized tracker already has. The benefit comes from randomizing beyond it, so the range should be set from the robot's measured latency, and delay margin belongs next to other sim-to-sim metrics.

Giovanni Minelli, Antonio Langella, Giovanni B. Dessy, ... , Victor Barasuol, Claudio Semini
总结: 将VLA模型部署到腿式操作机器人实现闭环loco-manipulation。
方法: 调整数据收集、模型适应与推理以协调基座姿态与臂。
证据: 在Unitree Go2加臂上实现可行的真机闭环部署。
为什么适合我: 腿足loco-manipulation,精准匹配核心研究兴趣。
推荐理由: 对应腿足loco-manipulation:把VLA部署到Go2加机械臂,需要同时协调底盘、身体姿态与末端,属于行走操作而非桌面臂。
原摘要

Vision-language-action (VLA) models are typically pretrained on broad, diverse datasets and subsequently fine-tuned on a comparatively small dataset tailored to a specific target robot. Such adaptation recipe is now widely established, achieving its most mature results on fixed-base arms due to their static base dynamics and consistent reference frame. Mounting the same arm on a quadruped extends the workspace to the entire floor, but fundamentally changes the embodiment: rather than controlling the arm in isolation, the policy must jointly coordinate base navigation, body posture, and end-effector motion. In this work, we study the practical demands of this embodiment change across data collection, model adaptation, and inference design to achieve closed-loop loco-manipulation on a Unitree Go2 with an AgileX Piper arm. We report the choices that made this real-robot deployment feasible, namely model variant, action horizon, and demonstration conventions, together with a study tailored to a pick-and-place task, separating data coverage from model capacity. We show that expanding demonstration coverage reduces held-out end-effector error by 30%, and incorporating secondary skills (such as pushing rather than grasping) introduces new behavioral modes without degrading baseline performance. Finally, closed-loop real-world experiments show how dataset composition shapes success and failure, validating the pipeline for more complex loco-manipulation.

Carson Kohlbrenner, Niraj Pudasaini, William Xie, ... , Nikolaus Correll, Alessandro Roncone
总结: 有损近距传感器作强偏置高效学习全身碰撞躲避。
方法: 强化学习测试多种表示,低分辨率有损传感器更优。
证据: 200多个人形在受限预算下端到端学会简化躲避球。
为什么适合我: 全身感知避碰,利于复杂地形接触丰富场景运动智能。
推荐理由: 在人形上用全身接近觉和强化学习学躲避,沾边全身控制,但不是运动跟踪、重定向、跑酷或loco-manipulation。
原摘要

Data-driven robots must learn to avoid collisions with as few demonstrations as possible to avoid collecting large datasets of destructive collision examples. We explore whole-body proximity sensors as inductive biases for learning reflexive collision-dodging behavior and find that lossy sensors with low resolution were stronger biases than high-resolution coverage. We also found that, perhaps unintuitively, anticipating when and where contact happens on the body of a robot reduced its learning sample efficiency. We investigated eight candidate representations in a reinforcement learning environment, which included latent compression, future contact anticipation, lossy corruption, and privileged state estimation. The candidate representations were tested by training over 200 humanoid robots to play a simplified version of dodgeball end-to-end under an intentionally constrained sample and compute budget.

Chubin Zhang, Changyuan Wang, Yirui Wang, ... , Tianle Zhang, Yansong Tang
总结: 联合机器人演示与噪声自我中心视频扩展策略学习。
方法: 视觉动作流结合可靠性条件腐蚀与加权回归处理噪声。
证据: 框架降低不可靠标签影响并利用结果作互补监督。
为什么适合我: 从人体视频扩展模仿学习,支持动作到全身策略。
推荐理由: 虽涉及自我中心视频的重建与重定向噪声,但主体是通用视觉-语言-动作策略,不是人形或腿足全身运动。
原摘要

Scaling policy learning requires broad experience and reliable supervision. Robot demonstrations provide high-quality trajectory supervision but are costly to collect. Egocentric videos broaden task and environment coverage, yet motion reconstruction and retargeting introduce noisy action labels. To address this challenge, we introduce EgoForesight, a vision-language-action framework that learns jointly from both sources by reducing the influence of unreliable action labels and exploiting demonstrated outcomes as complementary supervision. A joint vision-action flow couples future visual representation prediction with action generation, unifying reliability-aware action learning with outcome-informed representation transfer. Specifically, reliability-conditioned corruption allocates action supervision across noise levels according to source reliability, while reliability-weighted regression regulates each source's contribution. Complementing this treatment of noisy labels, privileged-future representation distillation draws supervision from demonstrated outcomes. An exponential-moving-average teacher observes the current scene and clean future to encode action-relevant structure from the resulting scene change. Its deeper representations supervise an earlier student layer receiving a corrupted future, providing outcome-informed structure beyond noisy action annotations. Together, these mechanisms support effective scaling with noisy egocentric actions. Over the tested range, EgoForesight exhibits steeper scaling slopes than the baseline, reducing real-world open-loop action MSE by 20.5% and improving RoboCasa few-shot success by 8.1 percentage points at 30,000 hours of pretraining data. After post-training, it also achieves strong absolute performance in both simulated and real-world dexterous manipulation.

Minho Park, Kinam Kim, Donghu Kim, ... , Hojoon Lee, Jaegul Choo
总结: 用强化学习作数据生成器发现多样成功演示供模仿。
方法: 潜条件RL生成器加反向课程与成功门控多样性目标。
证据: 四操作任务中拓宽成功状态覆盖并提升下游策略。
为什么适合我: 多样演示生成利于模仿学习全身运动与操作。
推荐理由: 用强化学习生成多样演示再做模仿,任务是通用操作,没有人形全身跟踪、重定向或腿足运动。
原摘要

Behavior cloning for robotic manipulation is highly sensitive to the coverage of its training data, yet collecting diverse successful demonstrations remains costly. We study reinforcement learning (RL) not as a deployable controller, but as a data generator for downstream visuomotor imitation learning. Our key premise is that multiple successful solution modes cover a broader range of successful trajectories than repeatedly collecting a single narrow solution. We introduce DiscoDemo, which trains a privileged latent-conditioned RL generator with a parallel reverse curriculum and a success-gated diversity objective to discover diverse solutions while preserving task success. The trained generator produces multiple successful closed-loop behaviors, which are safety-filtered and exported as a static dataset for downstream imitation learning. Across four manipulation tasks, DiscoDemo produces broader successful-state coverage and improves downstream visuomotor policies in both simulation and the real world. Averaged across tasks, downstream success increases to 62% from 40% for the strongest baseline in simulation, and to 78% from 39% on the real robot, respectively. These results suggest that successful solution diversity improves imitation learning for a pretrained flow-matching VLA and remains complementary to corrective data collection. Project page: https://davian-robotics.github.io/DiscoDemo

Regularised RL improves sim-to-real transfer

2.0/5 偏低 裁判分 6.0 Good 当日相对
Danilo Vucetic, Mohamad H. Danesh, Florence Clerc, Hsiu-Chin Lin, Jonathan Tremblay
总结: 正则化强化学习改善仿真到真机策略迁移性能。
方法: 提出基于软mellowmax值函数的SMM-AC算法。
证据: 真机更高回报且对观测环境动力学扰动更鲁棒。
为什么适合我: 提升RL sim-to-real,支持人形全身控制真机落地。
推荐理由: 讨论正则化强化学习的sim-to-real,方法相关但未聚焦人形全身跟踪、跑酷或loco-manipulation。
原摘要

Regularised reinforcement learning (RL) has provable robustness guarantees that should ease the transfer of policies from simulation to robots, known as sim-to-real. Yet, weakly regularised algorithms like Proximal Policy Optimisation (PPO) remain the de facto choice in sim-to-real pipelines, while the practical benefits of regularisation for transfer remain poorly characterised. In this work, we study the sim-to-real performance of commonly used RL algorithms, with the aim of characterising the effects of regularisation on robotic performance. To that end, we derive Soft MellowMax Actor-Critic (SMM-AC), a novel regularised RL algorithm based on the soft mellowmax value function, which carries strong robustness guarantees. On MuJoCo continuous control benchmarks, SMM-AC is competitive with baselines like Soft Actor-Critic (SAC). On real robots, SMM-AC achieves higher return, while exhibiting greater robustness to perturbations in observations, the environment, and dynamics than baselines. Together, these results demonstrate that regularised RL improves real-world policy deployment through better robustness to the sim-to-real gap, and changing real-world conditions.