NVIDIA · embodied AI & robot learning英伟达 · 具身智能与机器人学习
48 curated papers, code, datasets and hardware from NVIDIA (33 with code), with one-line takeaways.48 个来自英伟达的精选论文、代码、数据集与硬件（33 个附代码），均配一句话要点。
Labs are matched from each entry's affiliation field by simple name rules (data/labs.yaml); joint work appears under every lab named. Only curated entries are listed, because daily arXiv papers carry no affiliations.机构由每个条目的机构字段按名称规则匹配（data/labs.yaml），合作工作会出现在所有署名机构下。仅列出精选条目，因为每日 arXiv 论文不含机构信息。
GPU-parallel motion generation for arms — many trajectory optimizations run at once, giving collision-free, smooth plans in milliseconds.面向机械臂的 GPU 并行运动生成：同时运行大量轨迹优化，在毫秒级给出无碰撞且平滑的轨迹。
@misc{sundaralingam2023curobo,
title = {{cuRobo: Parallelized Collision-Free Minimum-Jerk Robot Motion Generation}},
author = {Balakumar Sundaralingam and Siva Kumar Sastry Hari and Adam Fishman and Caelan Garrett and Karl Van Wyk and Valts Blukis and Alexander Millane and Helen Oleynikova and Ankur Handa and Fabio Ramos and Nathan Ratliff and Dieter Fox},
year = {2023},
eprint = {2310.17274},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2310.17274},
}
Thousands of parallel GPU environments train a quadruped walking policy in minutes; legged_gym became a standard starter.数千个 GPU 并行环境几分钟就能训练出四足行走策略；legged_gym 成为标准入门框架。
@misc{rudin2021learning,
title = {{Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning}},
author = {Nikita Rudin and David Hoeller and Philipp Reist and Marco Hutter},
year = {2021},
eprint = {2109.11978},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2109.11978},
}
A family of omnimodal models that handles language, image, video, audio and action in one mixture-of-transformers, folding VLMs, video generators, world simulators and world-action models into a single framework.全模态世界模型家族，用一个 Mixture-of-Transformers 同时处理语言、图像、视频、音频和动作，把 VLM、视频生成、世界模拟器和世界动作模型并入同一框架。
@misc{nvidia2026cosmos,
title = {{Cosmos 3: Omnimodal World Models for Physical AI}},
author = {NVIDIA and : and Aditi and Niket Agarwal and Arslan Ali and Jon Allen and Martin Antolini and Adeline Aubame and Alisson Azzolini and Junjie Bai and Maciej Bala and Yogesh Balaji and Josh Bapst and Aarti Basant and Mukesh Beladiya and Mohammad Qazim Bhat and Zaid Pervaiz Bhat and Dan Blick and Vanni Brighella and Han Cai and Tiffany Cai and Eric Cameracci and Jiaxin Cao and Yulong Cao and Mark Carlson and Carlos Casanova and Ting-Yun Chang and Yan Chang and Yu-Wei Chao and Prithvijit Chattopadhyay and Roshan Chaudhari and Chieh-Yun Chen and Junyu Chen and Ke Chen and Qizhi Chen and Wenkai Chen and Xiaotong Chen and Yu Chen and An-Chieh Cheng and Click Cheng and Xiu Chia and Jeana Choi and Chaeyeon Chung and Wenyan Cong and Yin Cui and Magdalena Dadela and Nalin Dadhich and Wenliang Dai and Joyjit Daw and Alperen Degirmenci and others},
year = {2026},
eprint = {2606.02800},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2606.02800},
}
A foundation world model pretrained on 44k hours of egocentric human video, using continuous latent actions as proxies for missing labels, then post-trained on small robot datasets.在 4.4 万小时第一视角人类视频上预训练的基础世界模型，用连续潜动作代替缺失的动作标签，再用小规模机器人数据后训练。
@misc{gao2026dreamdojo,
title = {{DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos}},
author = {Shenyuan Gao and William Liang and Kaiyuan Zheng and Ayaan Malik and Seonghyeon Ye and Sihyun Yu and Wei-Cheng Tseng and Yuzhu Dong and Kaichun Mo and Chen-Hsuan Lin and Qianli Ma and Seungjun Nah and Loic Magne and Jiannan Xiang and Yuqi Xie and Ruijie Zheng and Dantong Niu and You Liang Tan and K. R. Zentner and George Kurian and Suneel Indupuru and Pooya Jannaty and Jinwei Gu and Jun Zhang and Jitendra Malik and Pieter Abbeel and Ming-Yu Liu and Yuke Zhu and Joel Jang and Linxi "Jim" Fan},
year = {2026},
eprint = {2602.06949},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2602.06949},
}
World Models世界模型Dexterous hand灵巧手human-videodexterouslatent-action
Tests scaling on humanoid control: more network, data and compute yield a generalist motion-tracking foundation model, with no manual reward engineering.检验规模定律在人形控制上的效果：扩大网络、数据和算力，得到通用动作跟踪基础模型，无需手工设计奖励。
@misc{luo2025sonic,
title = {{SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control}},
author = {Zhengyi Luo and Ye Yuan and Tingwu Wang and Chenran Li and Fernando Castañeda and Sirui Chen and Zi-Ang Cao and Jiefeng Li and David Minor and Qingwei Ben and Jinhyung Park and David Sami and Zi Wang and Xingye Da and Runyu Ding and Cyrus Hogg and Lina Song and Edy Lim and Eugene Jeong and Tairan He and Haoru Xue and Wenli Xiao and Simon Yuen and Jan Kautz and Yan Chang and Umar Iqbal and Linxi "Jim" Fan and Yuke Zhu},
year = {2025},
eprint = {2511.07820},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2511.07820},
}
Adapts Cosmos models for real-time interactive simulation: streams observations in response to actions, or renders them from an external simulator's state. Shown on manipulation, video transfer and multi-view AV.将 Cosmos 模型用于实时交互仿真的统一框架：根据动作流式生成观测，或依据外部仿真器的物理状态生成观测。示例涵盖操作、可控视频迁移和多视角自动驾驶。
@misc{nvidia2026interactive,
title = {{Interactive World Simulation with Cosmos for Physical AI}},
author = {NVIDIA and : and Anis Ahmad and Jon Allen and Malik Aqeel Anwar and Jesse Archer and Maciej Bala and Piotr Barejko and Aarti Basant and Dan Blick and Taras Bolzhelarskyi and Henry Che and Jin Chen and Yukang Chen and Tae Eun Choe and Yuzhu Dong and Nat Duca and Liang Feng and Francesco Ferroni and Xiao Fu and Mohit Gambhir and Vivek Goel and Rony Goldenthal and Aryaman Gupta and Ali Hassani and Nathan Hayes-Roth and Yong He and Michael Honke and Shengyu Huang and Wei Huang and DeLesley Hutchins and Suneel Indupuru and Pooya Jannaty and Jingyi Jin and Janmesh Kalra and Ashna Khetan and Xin Kong and Aleksandr Korovko and Hui Kuang and Zhengfei Kuang and George Kurian and Jean-Francois Lafleche and Jayjun Lee and Justin Leung and William Lew and Xiang Li and Zhaoshuo Li and Chen-Hsuan Lin and Ming-Yu Liu and Xiangyu Lu and others},
year = {2026},
eprint = {2611.06627},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2611.06627},
}
Commercially licensed 3B-parameter humanoid VLA: a Cosmos-Reason2 backbone plans and a diffusion transformer outputs motor commands. NVIDIA reports training on about 20,854 hours of human egocentric video.可商用的 30 亿参数人形 VLA：Cosmos-Reason2 主干负责推理规划，扩散 Transformer 输出电机指令。NVIDIA 称用约 20,854 小时人类第一视角视频训练。
VLA that reasons in discrete motion tokens (a chain of dynamics) shared across embodiments; predicted futures are scored against what actually happened, so action-free human videos give RL signal. Sim and real tests.VLA 用跨本体共享的离散运动 token 链（Chain-of-Dynamics）表达动作思考；预测每条链导致的未来并与真实结果比对来强化，使没有动作标注的人类视频也能提供监督。在仿真与真机上验证人到机器人的迁移。
@misc{kim2026thinkworld,
title = {{ThinkWorld: World Imagination Reinforces Chain-of-Dynamics Reasoning}},
author = {Chaehyun Kim and Chi-Pin Huang and Jisoo Kim and Min-Hung Chen and Yu-Chiang Frank Wang and Seungryong Kim and Fu-En Yang},
year = {2026},
eprint = {2610.37666},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2610.37666},
}
VLMs plan with points, boxes and trajectories one coordinate token at a time; VFlash drafts a whole primitive in one pass. Reports lower trajectory error, ~25x faster 3D point forecasting, a WAM gain (62.6% → 70.2%).用点、框、轨迹做视觉规划的视觉语言模型需要逐个坐标 token 解码；VFlash 用轻量草稿器一次前向生成整个图元。报告轨迹误差更低、三维点运动预测约快 25 倍，接入世界动作模型后成功率 62.6% → 70.2%。
@misc{chou2026vflash,
title = {{VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding}},
author = {Yu-Cheng Chou and Shihao Wang and Jin Huang and Chi-Pin Huang and Chiao-An Yang and Xianzhe Fan and Jing Wang and Zhiqi Li and Yuanguo Kuang and Ruijie Zheng and Johan Bjorck and Sifei Liu and Hang Su and Abhishek Badki and Lei Zhang and Alan Yuille and Linxi "Jim'' Fan and Jan Kautz and Zhiding Yu},
year = {2026},
eprint = {2611.08449},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2611.08449},
}
Learns a residual "delta action" model from real rollouts to fix sim physics, enabling agile athletic humanoid motions.从真实轨迹学习残差"增量动作"模型来修正仿真物理，实现敏捷的人形运动技能。
@misc{he2025asap,
title = {{ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skills}},
author = {Tairan He and Jiawei Gao and Wenli Xiao and Yuanhang Zhang and Zi Wang and Jiashun Wang and Zhengyi Luo and Guanqi He and Nikhil Sobanbab and Chaoyi Pan and Zeji Yi and Guannan Qu and Kris Kitani and Jessica Hodgins and Linxi "Jim" Fan and Yuke Zhu and Changliu Liu and Guanya Shi},
year = {2025},
eprint = {2502.01143},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2502.01143},
}
Stereo depth model trained on large synthetic data that reports strong zero-shot accuracy on real scenes — a drop-in depth source for stereo-camera robots.在大规模合成数据上训练的双目深度模型，报告称在真实场景上零样本精度很强；可直接作为双目相机机器人的深度来源。
@misc{wen2025foundationstereo,
title = {{FoundationStereo: Zero-Shot Stereo Matching}},
author = {Bowen Wen and Matthew Trepte and Joseph Aribido and Jan Kautz and Orazio Gallo and Stan Birchfield},
year = {2025},
eprint = {2501.09898},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2501.09898},
}
@misc{nvidia2025gr,
title = {{GR00T N1: An Open Foundation Model for Generalist Humanoid Robots}},
author = {NVIDIA and : and Johan Bjorck and Fernando Castañeda and Nikita Cherniadev and Xingye Da and Runyu Ding and Linxi "Jim" Fan and Yu Fang and Dieter Fox and Fengyuan Hu and Spencer Huang and Joel Jang and Zhenyu Jiang and Jan Kautz and Kaushil Kundalia and Lawrence Lao and Zhiqi Li and Zongyu Lin and Kevin Lin and Guilin Liu and Edith Llontop and Loic Magne and Ajay Mandlekar and Avnish Narayan and Soroush Nasiriany and Scott Reed and You Liang Tan and Guanzhi Wang and Zu Wang and Jing Wang and Qi Wang and Jiannan Xiang and Yuqi Xie and Yinzhen Xu and Zhenjia Xu and Seonghyeon Ye and Zhiding Yu and Ao Zhang and Hao Zhang and Yizhou Zhao and Ruijie Zheng and Yuke Zhu},
year = {2025},
eprint = {2503.14734},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2503.14734},
}
The reference paper for Isaac Lab, Orbit's successor — GPU-parallel physics, rendering and sensor simulation with ready-made robots and tasks for RL and imitation.Isaac Lab（Orbit 的后继）的正式论文：GPU 并行的物理、渲染与传感器仿真，内置机器人与任务，面向强化学习和模仿学习。
@misc{nvidia2025isaac,
title = {{Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learning}},
author = {NVIDIA and : and Mayank Mittal and Pascal Roth and James Tigue and Antoine Richard and Octi Zhang and Peter Du and Antonio Serrano-Muñoz and Xinjie Yao and René Zurbrügg and Nikita Rudin and Lukasz Wawrzyniak and Milad Rakhsha and Alain Denzler and Eric Heiden and Ales Borovicka and Ossama Ahmed and Iretiayo Akinola and Abrar Anwar and Mark T. Carlson and Ji Yuan Feng and Animesh Garg and Renato Gasoto and Lionel Gulich and Yijie Guo and M. Gussert and Alex Hansen and Mihir Kulkarni and Chenran Li and Wei Liu and Viktor Makoviychuk and Grzegorz Malczyk and Hammad Mazhar and Masoud Moghani and Adithyavairavan Murali and Michael Noseworthy and Alexander Poddubny and Nathan Ratliff and Welf Rehberg and Clemens Schwarke and Ritvik Singh and James Latham Smith and Bingjie Tang and Ruchik Thaker and Matthew Trepte and Karl Van Wyk and Fangzhou Yu and Alex Millane and Vikram Ramasamy and others},
year = {2025},
eprint = {2511.04831},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2511.04831},
}
Open, GPU-accelerated physics engine built on NVIDIA Warp and OpenUSD with multiple solvers for contact-rich robot learning; co-developed by Disney Research, Google DeepMind and NVIDIA, hosted by the Linux Foundation.基于 NVIDIA Warp 和 OpenUSD 的开源 GPU 物理引擎，支持多种求解器，面向接触丰富的机器人学习；由 Disney Research、Google DeepMind 和 NVIDIA 联合开发，由 Linux 基金会托管。
A flow-based world foundation model unifying Text2World, Image2World and Video2World, released at 2B and 14B, with Cosmos-Transfer2.5 for Sim2Real translation.基于流的世界基础模型，统一 Text2World、Image2World 和 Video2World，发布 2B 和 14B 两种规模，并配有用于 Sim2Real 转换的 Cosmos-Transfer2.5。
@misc{nvidia2025world,
title = {{World Simulation with Video Foundation Models for Physical AI}},
author = {NVIDIA and : and Arslan Ali and Junjie Bai and Maciej Bala and Yogesh Balaji and Aaron Blakeman and Tiffany Cai and Jiaxin Cao and Tianshi Cao and Elizabeth Cha and Yu-Wei Chao and Prithvijit Chattopadhyay and Mike Chen and Yongxin Chen and Yu Chen and Shuai Cheng and Yin Cui and Jenna Diamond and Yifan Ding and Jiaojiao Fan and Linxi Fan and Liang Feng and Francesco Ferroni and Sanja Fidler and Xiao Fu and Ruiyuan Gao and Yunhao Ge and Jinwei Gu and Aryaman Gupta and Siddharth Gururani and Imad El Hanafi and Ali Hassani and Zekun Hao and Jacob Huffman and Joel Jang and Pooya Jannaty and Jan Kautz and Grace Lam and Xuan Li and Zhaoshuo Li and Maosheng Liao and Chen-Hsuan Lin and Tsung-Yi Lin and Yen-Chen Lin and Huan Ling and Ming-Yu Liu and Xian Liu and Yifan Lu and Alice Luo and others},
year = {2025},
eprint = {2511.00062},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2511.00062},
}
120 kitchen scenes, 2,500+ objects and 100 tasks with generated demonstrations for training generalist policies.120 个厨房场景、2500 多个物体与 100 个任务，并附生成演示，用于训练通用策略。
@misc{nasiriany2024robocasa,
title = {{RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots}},
author = {Soroush Nasiriany and Abhiram Maddukuri and Lance Zhang and Adeet Parikh and Aaron Lo and Abhishek Joshi and Ajay Mandlekar and Yuke Zhu},
year = {2024},
eprint = {2406.02523},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2406.02523},
}
Tracks the 6-DoF pose of a never-seen object in RGB-D video while building its neural 3D model online, with no CAD model needed; useful for manipulating novel objects.在 RGB-D 视频中跟踪从未见过物体的六自由度位姿，同时在线构建其神经三维模型，无需 CAD 模型；适用于操作新物体。
@misc{wen2023bundlesdf,
title = {{BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects}},
author = {Bowen Wen and Jonathan Tremblay and Valts Blukis and Stephen Tyree and Thomas Muller and Alex Evans and Dieter Fox and Jan Kautz and Stan Birchfield},
year = {2023},
eprint = {2303.14158},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2303.14158},
}
6D pose estimation and tracking of unseen objects from a CAD model or a few reference images, with no per-object training.只需 CAD 模型或少量参考图像即可对未见过的物体做 6D 位姿估计与跟踪，无需逐物体训练。
@misc{wen2023foundationpose,
title = {{FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects}},
author = {Bowen Wen and Wei Yang and Jan Kautz and Stan Birchfield},
year = {2023},
eprint = {2312.08344},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2312.08344},
}
@misc{mittal2023orbit,
title = {{Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments}},
author = {Mayank Mittal and Calvin Yu and Qinxi Yu and Jingzhou Liu and Nikita Rudin and David Hoeller and Jia Lin Yuan and Ritvik Singh and Yunrong Guo and Hammad Mazhar and Ajay Mandlekar and Buck Babich and Gavriel State and Marco Hutter and Animesh Garg},
year = {2023},
eprint = {2301.04195},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2301.04195},
}
Learns reusable skill embeddings for simulated characters by combining adversarial imitation and unsupervised RL, so new tasks need not start from scratch.结合对抗模仿与无监督强化学习，为仿真角色学习可复用的技能嵌入，新任务不必从零训练。
Continual off-policy RL for VLAs — small actor/critic networks carry over between tasks, distilled on a few kept episodes to avoid forgetting; reports 31–68.5% less real-robot time on new assembly tasks.面向 VLA 的持续离策略强化学习：轻量 actor/critic 在任务间延续，用少量保留的旧任务轨迹做蒸馏以防遗忘。论文报告新的接触式装配任务所需真机训练时间减少 31–68.5%，旧任务几乎不遗忘。
@misc{kim2026accrue,
title = {{ACCRUE: Continual Off-Policy Reinforcement Learning for Robot Foundation Models}},
author = {Changyeon Kim and Kimin Lee and Yashraj Narang and Yijie Guo},
year = {2026},
eprint = {2610.18596},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2610.18596},
}
The open data engine behind GR00T training: models that write timestamped action captions, dense process rewards and episode outcome rewards for human and robot video; reports SOTA on WGO-Bench, RoboReward and Robometer.GR00T 训练所用的开放数据引擎：一组模型为人类与机器人视频生成带时间戳的动作描述、稠密过程奖励和整段结果奖励；报告在 WGO-Bench、RoboReward 与 Robometer 上达到最优。
@misc{leng2026flora,
title = {{FLORA: An Open Model Suite for Scaling Robotics Data Engines}},
author = {Sicong Leng and Yunze Man and Zhiqi Li and Jing Wang and Yuhe Zhang and Ryan Punamiya and Siddharth Patki and Kaiyuan Zheng and Zi-ang Cao and Zhiding Yu and Sifei Liu and Danfei Xu and Ben Levin and Jan Kautz and Yuke Zhu and Johan Bjorck and Linxi "Jim" Fan},
year = {2026},
eprint = {2610.31779},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2610.31779},
}
CoRL 2026 Oral. A fully virtual pipeline for humanoid loco-manipulation data: grounds video world models in 3D assets and simulator-ready scenes, with no teleoperation or physical setups.CoRL 2026 Oral。全虚拟的人形移动操作数据管线：把视频世界模型落到 3D 资产与可仿真场景中，无需遥操作或搭建实体环境。
@misc{xie2026grail,
title = {{GRAIL: Generating Humanoid Loco-Manipulation from World Models}},
author = {Tianyi Xie and Haotian Zhang and Jinhyung Park and Zi Wang and Bowen Wen and Jiefeng Li and Xueting Li and Qingwei Ben and Haoyang Weng and Yufei Ye and David Minor and Tingwu Wang and Chenfanfu Jiang and Sanja Fidler and Jan Kautz and Linxi Fan and Yuke Zhu and Zhengyi Luo and Umar Iqbal and Ye Yuan},
year = {2026},
eprint = {2606.05160},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2606.05160},
}
One frozen humanoid policy driven by prompts (timed body-part targets, velocity, style); skills are plain code, refined by coding agents. On a Unitree G1 it walks, recovers, picks up objects and sits.可“提示”的人形全身执行层：冻结的生成式基础策略接收部分目标（定时的身体部位目标、速度指令、风格）生成全身动作；技能是普通代码，可由编码智能体分析动作数据后自动改进，无需按任务重训。在 Unitree G1 上一个检查点即可行走、恢复、遥操作抓取和坐下。
@misc{tessler2026metatron,
title = {{METATRON: A Promptable Whole-Body Execution Layer for Humanoids}},
author = {Chen Tessler and Yifeng Jiang and Alec Otto and Yi Shi and Umar Iqbal and Gal Chechik and Xue Bin Peng},
year = {2026},
eprint = {2610.38027},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2610.38027},
}
@misc{jiang2026robottt,
title = {{RoboTTT: Context Scaling for Robot Policies}},
author = {Yunfan Jiang and Yevgen Chebotar and Ruijie Zheng and Fengyuan Hu and Yunhao Ge and Jimmy Wu and Tianyuan Dai and Scott Reed and Li Fei-Fei and Yuke Zhu and Linxi "Jim" Fan},
year = {2026},
eprint = {2607.15275},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2607.15275},
}
@misc{nvidia2025cosmos,
title = {{Cosmos World Foundation Model Platform for Physical AI}},
author = {NVIDIA and : and Niket Agarwal and Arslan Ali and Maciej Bala and Yogesh Balaji and Erik Barker and Tiffany Cai and Prithvijit Chattopadhyay and Yongxin Chen and Yin Cui and Yifan Ding and Daniel Dworakowski and Jiaojiao Fan and Michele Fenzi and Francesco Ferroni and Sanja Fidler and Dieter Fox and Songwei Ge and Yunhao Ge and Jinwei Gu and Siddharth Gururani and Ethan He and Jiahui Huang and Jacob Huffman and Pooya Jannaty and Jingyi Jin and Seung Wook Kim and Gergely Klár and Grace Lam and Shiyi Lan and Laura Leal-Taixe and Anqi Li and Zhaoshuo Li and Chen-Hsuan Lin and Tsung-Yi Lin and Huan Ling and Ming-Yu Liu and Xian Liu and Alice Luo and Qianli Ma and Hanzi Mao and Kaichun Mo and Arsalan Mousavian and Seungjun Nah and Sriharsha Niverty and David Page and Despoina Paschalidou and Zeeshan Patel and Lindsey Pavao and others},
year = {2025},
eprint = {2501.03575},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2501.03575},
}
DexUMI — an exoskeleton matched to the robot hand's fingertip workspace, plus inpainting that swaps the human hand for the robot's; reports 86% average success (Inspire, XHand) and ~3.2× faster collection than teleop.DexUMI：按目标机器人手的指尖工作空间设计的可穿戴手部外骨骼，再用视频修补把画面中的人手换成机器人手；论文报告在 Inspire 与 XHand 上平均成功率 86%，采集效率约为遥操作的 3.2 倍。
@misc{xu2025dexumi,
title = {{DexUMI: Using Human Hand as the Universal Manipulation Interface for Dexterous Manipulation}},
author = {Mengda Xu and Han Zhang and Yifan Hou and Zhenjia Xu and Linxi Fan and Manuela Veloso and Shuran Song},
year = {2025},
eprint = {2505.21864},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2505.21864},
}
Fine-tunes video world models on a robot to dream new demos, labels them with latent actions or an inverse dynamics model, and trains policies on them; a humanoid learns 20+ new behaviors from one-scene teleop data.把视频世界模型微调到目标机器人上“做梦”生成新示范，再用隐动作或逆动力学模型补上动作标签用于训练策略；人形机器人仅凭单一场景的遥操作数据学会 20 多种新行为。
@misc{jang2025dreamgen,
title = {{DreamGen: Unlocking Generalization in Robot Learning through Video World Models}},
author = {Joel Jang and Seonghyeon Ye and Zongyu Lin and Jiannan Xiang and Johan Bjorck and Yu Fang and Fengyuan Hu and Spencer Huang and Kaushil Kundalia and Yen-Chen Lin and Loic Magne and Ajay Mandlekar and Avnish Narayan and You Liang Tan and Guanzhi Wang and Jing Wang and Qi Wang and Yinzhen Xu and Xiaohui Zeng and Kaiyuan Zheng and Ruijie Zheng and Ming-Yu Liu and Luke Zettlemoyer and Dieter Fox and Jan Kautz and Scott Reed and Yuke Zhu and Linxi Fan},
year = {2025},
eprint = {2505.12705},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2505.12705},
}
Adds a few tokens to a VLA-style diffusion transformer and aligns them with latents of future observations, so the policy anticipates the future without a separate world model.只在 VLA 式扩散 Transformer 中加入少量 token，并让它们对齐未来观测的潜表示，使策略无需独立世界模型就能预判未来。
@misc{zheng2025flare,
title = {{FLARE: Robot Learning with Implicit World Modeling}},
author = {Ruijie Zheng and Jing Wang and Scott Reed and Johan Bjorck and Yu Fang and Fengyuan Hu and Joel Jang and Kaushil Kundalia and Zongyu Lin and Loic Magne and Avnish Narayan and You Liang Tan and Guanzhi Wang and Qi Wang and Jiannan Xiang and Yinzhen Xu and Seonghyeon Ye and Jan Kautz and Furong Huang and Yuke Zhu and Linxi Fan},
year = {2025},
eprint = {2505.15659},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2505.15659},
}
A hierarchical VLA in which a fine-tuned VLM draws a coarse 2D path for the end effector on the image and a low-level 3D policy follows it, so the VLM can be trained on cheap off-domain data such as sim or videos.分层 VLA：微调后的 VLM 在图像上画出末端执行器的粗略二维路径，再由底层三维策略跟随执行；因此 VLM 可以用仿真、视频等低成本的非本域数据训练。
@misc{li2025hamster,
title = {{HAMSTER: Hierarchical Action Models for Open-World Robot Manipulation}},
author = {Yi Li and Yuquan Deng and Jesse Zhang and Joel Jang and Marius Memmel and Raymond Yu and Caelan Reed Garrett and Fabio Ramos and Dieter Fox and Anqi Li and Abhishek Gupta and Ankit Goyal},
year = {2025},
eprint = {2502.05485},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2502.05485},
}
Extends MimicGen to humanoids with two dexterous hands, turning 60 human demos into 21K simulated ones across bimanual tasks, plus a real-to-sim-to-real pipeline tested on real can sorting.把 MimicGen 扩展到双灵巧手人形机器人，从 60 条人类演示生成 2.1 万条双臂仿真演示，并给出在真实分拣罐子任务上验证的 real-to-sim-to-real 流程。
@misc{jiang2024dexmimicgen,
title = {{DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning}},
author = {Zhenyu Jiang and Yuqi Xie and Kevin Lin and Zhenjia Xu and Weikang Wan and Ajay Mandlekar and Linxi Fan and Yuke Zhu},
year = {2024},
eprint = {2410.24185},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2410.24185},
}
Distills one policy that supports several humanoid control modes (root velocity, joint angles, keypoint tracking) instead of training a separate controller per mode.蒸馏出单一策略，支持多种人形控制模式（根部速度、关节角、关键点跟踪），不必为每种模式各训练一个控制器。
@misc{he2024hover,
title = {{HOVER: Versatile Neural Whole-Body Controller for Humanoid Robots}},
author = {Tairan He and Wenli Xiao and Toru Lin and Zhengyi Luo and Zhenjia Xu and Zhenyu Jiang and Jan Kautz and Changliu Liu and Guanya Shi and Xiaolong Wang and Linxi Fan and Yuke Zhu},
year = {2024},
eprint = {2410.21229},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2410.21229},
}
LAPA pretrains a VLA without action labels: it learns discrete latent actions between video frames, predicts them from image + text, then maps them to robot actions; reports beating OpenVLA on real tasks.LAPA 无需动作标注即可预训练 VLA：先从视频相邻帧间学习离散隐动作，再训练模型由图像和指令预测隐动作，最后用少量机器人数据映射到真实动作；报告在真机任务上优于 OpenVLA。
@misc{ye2024latent,
title = {{Latent Action Pretraining from Videos}},
author = {Seonghyeon Ye and Joel Jang and Byeongguk Jeon and Sejune Joo and Jianwei Yang and Baolin Peng and Ajay Mandlekar and Reuben Tan and Yu-Wei Chao and Bill Yuchen Lin and Lars Liden and Kimin Lee and Jianfeng Gao and Luke Zettlemoyer and Dieter Fox and Minjoon Seo},
year = {2024},
eprint = {2410.11758},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2410.11758},
}
A VLA outputs mid-level navigation commands in language that an RL locomotion policy executes, so legged robots follow natural-language routes; adds Isaac Lab VLN benchmarks.VLA 先用语言输出中层导航指令，再由强化学习运动策略执行，让腿式机器人按自然语言指令导航；并提供 Isaac Lab 上的 VLN 基准。
@misc{cheng2024navila,
title = {{NaVILA: Legged Robot Vision-Language-Action Model for Navigation}},
author = {An-Chieh Cheng and Yandong Ji and Zhaojing Yang and Zaitian Gongye and Xueyan Zou and Jan Kautz and Erdem Bıyık and Hongxu Yin and Sifei Liu and Xiaolong Wang},
year = {2024},
eprint = {2412.04453},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2412.04453},
}
@misc{pumacay2024colosseum,
title = {{THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation}},
author = {Wilbert Pumacay and Ishika Singh and Jiafei Duan and Ranjay Krishna and Jesse Thomason and Dieter Fox},
year = {2024},
eprint = {2402.08191},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2402.08191},
}
GPU-based simulation of GelSight-style tactile images and contact forces in Isaac Gym, plus learning tools for sim-to-real transfer of visuotactile policies.在 Isaac Gym 中用 GPU 模拟 GelSight 式触觉图像与接触力，并提供用于视触觉策略仿真到真实迁移的学习工具。
@misc{akinola2024tacsl,
title = {{TacSL: A Library for Visuotactile Sensor Simulation and Learning}},
author = {Iretiayo Akinola and Jie Xu and Jan Carius and Dieter Fox and Yashraj Narang},
year = {2024},
eprint = {2408.06506},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2408.06506},
}
A learned navigation policy sequences pre-trained skills (walk, jump, climb, crouch) using a learned obstacle reconstruction; reports up to 2 m/s on hardware (Science Robotics).分层的全学习系统：导航策略结合学习到的障碍物重建来调度预训练技能（行走、跳跃、攀爬、下蹲）；报告实机速度可达 2 米/秒。发表于 Science Robotics。
Camera-only teleoperation for arm + dexterous hand pairs - tracks the operator's hand and retargets it to many different robot hands, in sim or on real hardware.仅用相机即可遥操作"机械臂 + 灵巧手"：跟踪操作者手部并重定向到多种不同的机器人手，可用于仿真或真机。
@misc{qin2023anyteleop,
title = {{AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System}},
author = {Yuzhe Qin and Wei Yang and Binghao Huang and Karl Van Wyk and Hao Su and Xiaolong Wang and Yu-Wei Chao and Dieter Fox},
year = {2023},
eprint = {2307.04577},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2307.04577},
}
Turns a handful of human demos into thousands by splitting them into object-centric segments and replaying those segments in new scenes; widely reused to scale imitation data.把少量人类演示切分为以物体为中心的片段，并在新场景中变换重放，从而扩增为数千条演示；被广泛用于扩充模仿学习数据。
@misc{mandlekar2023mimicgen,
title = {{MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations}},
author = {Ajay Mandlekar and Soroush Nasiriany and Bowen Wen and Iretiayo Akinola and Yashraj Narang and Linxi Fan and Yuke Zhu and Dieter Fox},
year = {2023},
eprint = {2310.17596},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2310.17596},
}
RVT — re-renders the point cloud into a few virtual views and uses a multi-view Transformer to predict keyframe poses; reported to train and run much faster than voxel-based PerAct on RLBench. Repo also hosts RVT-2.RVT：把点云重新渲染为若干虚拟视角，再用多视角 Transformer 预测关键帧位姿；论文报告在 RLBench 上训练与推理都比基于体素的 PerAct 快得多。代码库同时包含 RVT-2。
@misc{goyal2023rvt,
title = {{RVT: Robotic View Transformer for 3D Object Manipulation}},
author = {Ankit Goyal and Jie Xu and Yijie Guo and Valts Blukis and Yu-Wei Chao and Dieter Fox},
year = {2023},
eprint = {2306.14896},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2306.14896},
}
A neural policy trained on millions of planner-generated trajectories that moves an arm to goals straight from point clouds.在数百万条规划器生成的轨迹上训练的神经策略，可直接根据点云把机械臂移动到目标位置。
PerAct — a Perceiver Transformer over voxelized RGB-D observations that predicts the next keyframe gripper pose as voxel classification; one language-conditioned policy for many RLBench and real tasks.PerAct：在体素化的 RGB-D 观测上运行 Perceiver Transformer，把下一关键帧夹爪位姿当作体素分类来预测；一个语言条件策略即可覆盖众多 RLBench 与真机任务。
Casts many tabletop tasks as interleaved text-and-image prompts to one Transformer policy, with the VIMA-Bench suite for systematic generalization tests.把多种桌面任务统一成图文交错的提示，交给一个 Transformer 策略执行，并配套 VIMA-Bench 用于系统性泛化评测。
@misc{jiang2022vima,
title = {{VIMA: General Robot Manipulation with Multimodal Prompts}},
author = {Yunfan Jiang and Agrim Gupta and Zichen Zhang and Guanzhi Wang and Yongqiang Dou and Yanjun Chen and Li Fei-Fei and Anima Anandkumar and Yuke Zhu and Linxi Fan},
year = {2022},
eprint = {2210.03094},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2210.03094},
}
CLIPort — combines CLIP's semantic "what" stream with Transporter's spatial "where" stream for language-conditioned tabletop manipulation; an early bridge from web vision-language models to robot actions.CLIPort：把 CLIP 提供的语义"是什么"通路与 Transporter 的空间"在哪里"通路结合，实现语言条件的桌面操作；是较早把网络视觉语言模型接入机器人动作的工作。
@misc{shridhar2021cliport,
title = {{CLIPort: What and Where Pathways for Robotic Manipulation}},
author = {Mohit Shridhar and Lucas Manuelli and Dieter Fox},
year = {2021},
eprint = {2109.12098},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2109.12098},
}
Contact-GraspNet — predicts 6-DoF parallel-jaw grasps for a whole cluttered scene from one depth view by treating observed points as grasp contacts, which shrinks the problem to 4 DoF per point.Contact-GraspNet：从单帧深度图为整个杂乱场景预测 6 自由度平行夹爪抓取；把观测到的点当作抓取接触点，使每个点只需学习 4 个自由度。
@misc{chao2021dexycb,
title = {{DexYCB: A Benchmark for Capturing Hand Grasping of Objects}},
author = {Yu-Wei Chao and Wei Yang and Yu Xiang and Pavlo Molchanov and Ankur Handa and Jonathan Tremblay and Yashraj S. Narang and Karl Van Wyk and Umar Iqbal and Stan Birchfield and Jan Kautz and Dieter Fox},
year = {2021},
eprint = {2104.04631},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2104.04631},
}
@misc{makoviychuk2021isaac,
title = {{Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning}},
author = {Viktor Makoviychuk and Lukasz Wawrzyniak and Yunrong Guo and Michelle Lu and Kier Storey and Miles Macklin and David Hoeller and Nikita Rudin and Arthur Allshire and Ankur Handa and Gavriel State},
year = {2021},
eprint = {2108.10470},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2108.10470},
}
RMPs — describe each behaviour (reach, avoid obstacles, joint limits) as an acceleration policy plus a Riemannian metric, then combine them geometrically; the basis of later RMPflow and geometric fabrics controllers.RMP：把每种行为（到达、避障、关节限位）表示为加速度策略加黎曼度量，再按几何方式组合；是后来 RMPflow 与几何织物（geometric fabrics）控制器的基础。
@misc{ratliff2018riemannian,
title = {{Riemannian Motion Policies}},
author = {Nathan D. Ratliff and Jan Issac and Daniel Kappler and Stan Birchfield and Dieter Fox},
year = {2018},
eprint = {1801.02854},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/1801.02854},
}