Jiayuan Gu

I am an Assistant Professor at ShanghaiTech University. My research topics include Embodied AI and 3D Vision. Specifically, I am interested in developing robust robot learning systems in both simulation and the real world.

Prior to joining ShanghaiTech, I received my Ph.D. from Computer Science & Engineering (CSE) at UC San Diego, advised by Prof. Hao Su. I obtained my Bachelor degree at Peking University, advised by Prof. Liwei Wang. Besides, I spent good time in Google DeepMind (Robotics), Facebook AI (Embodied AI), Waymo, Uber ATG, and MSRA.

Prospective students: If you are interested in joining my group, please read this.

Email / Google Scholar / Github

News

[2025.6] Our work CAST receives the SIGGRAPH 2025 Best Paper Award.

[2025.5] Co-organize the workshop on Human-Robot-Scene at ICCV 2025.

[2024.4] I will join ShanghaiTech University as an Assistant Professor.

[2024.1] RT-Trajectory is accepted to ICLR 2024 as a spotlight (5%).

[2023.7] Co-organize the GMPL workshop at RSS 2023.

[2023.1] Two papers are accepted to ICLR 2023.

[2022.8] ManiSkill 2022 Challenge is open!

[2022.6] I will give a talk at the CVPR 2022 tutorial Building and Working in Environments for Embodied AI.

[2022.4] Co-organize the first VOLI workshop at ECCV 2022.

[2020.9] Two papers are accepted to NeurIPS 2020.

[2020.7] Two papers (spotlight) are accepted to ECCV 2020.

[2019.01.12] Codes for Multi-view PointNet for 3D Scene Understanding are released.

Selected Publications

To see the full list of my research papers, please visit my google scholar profile.

CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image

Kaixin Yao*, Longwen Zhang*, Xinhao Yan, Yan Zeng, Qixuan Zhang, Wei Yang, Lan Xu, Jiayuan Gu, Jingyi Yu

SIGGRAPH 2025 (Best Paper Award)

project website / arXiv

Point-SAM: Promptable 3D Segmentation Model for Point Clouds

Yuchen Zhou*, Jiayuan Gu*, Tung Yen Chiang, Fanbo Xiang, Hao Su

ICLR 2025

project website / arXiv / Code

Evaluating Real-World Robot Manipulation Policies in Simulation

Xuanlin Li*, Kyle Hsu*, Jiayuan Gu*, Karl Pertsch†, Oier Mees†, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su‡, Quan Vuong‡, Ted Xiao‡

CoRL 2024

project website / arXiv / Code

One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion

Minghua Liu*, Ruoxi Shi*, Linghao Chen*, Zhuoyang Zhang*, Chao Xu*, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, Hao Su

CVPR 2024

project website

RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches

Jiayuan Gu, Sean Kirmani, Paul Wohlhart, Yao Lu, Montserrat Gonzalez Arenas, Kanishka Rao, Wenhao Yu, Chuyuan Fu, Keerthana Gopalakrishnan, Zhuo Xu, Priya Sundaresan, Peng Xu, Hao Su, Karol Hausman, Chelsea Finn, Quan Vuong, Ted Xiao

ICLR 2024 (spotlight)

project website / arXiv / blog

RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches

Priya Sundaresan, Quan Vuong, Jiayuan Gu, Peng Xu, Ted Xiao, Sean Kirmani, Tianhe Yu, Michael Stark, Ajinkya Jain, Karol Hausman, Dorsa Sadigh*, Jeannette Bohg*, Stefan Schaal*

project website

An Open-source Recipe for Building Simulated Robot Manipulation Benchmarks

Jiayuan Gu, Linghao Chen, Zhiwei Jia, Fanbo Xiang, Hao Su

ICRA 2023 COMPARE workshop

pdf

ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills

Jiayuan Gu, Fanbo Xiang, et.al.

ICLR 2023

Challenge website / code / paper

Multi-skill Mobile Manipulation for Object Rearrangement

Jiayuan Gu, Devendra Singh Chaplot, Hao Su, Jitendra Malik

ICLR 2023 (spotlight)

project website / code / arxiv

Rank 1st in Habitat Rearrangement Challenge 2022

Compositionally Generalizable 3D Structure Prediction

Songfang Han, Jiayuan Gu, Kaichun Mo, Li Yi, Siyu Hu, Xuejin Chen, Hao Su

Preprint

code / arXiv

Refactoring Policy for Compositional Generalizability using Self-Supervised Object Proposals

Tongzhou Mu*, Jiayuan Gu*, Zhiwei Jia, Hao Tang, Hao Su

NeurIPS 2020

project website / code / arXiv

Towards Scale-Invariant Graph-related Problem Solving by Iterative Homogeneous Graph Neural Networks

Hao Tang, Zhiao Huang, Jiayuan Gu, Bao-Liang Lu, Hao Su

NeurIPS 2020

code / arXiv

Weakly-supervised 3D Shape Completion in the Wild

Jiayuan Gu, Wei-Chiu Ma, Sivabalan Manivasagam, Wenyuan Zeng, Zihao Wang, Yuwen Xiong, Hao Su, Raquel Urtasun

ECCV 2020 (Spotlight)

arXiv

Deep Feedback Inverse Problem Solver

Wei-Chiu Ma, Shenlong Wang, Jiayuan Gu, Sivabalan Manivasagam, Antonio Torralba, Raquel Urtasun

ECCV 2020 (Spotlight)

arXiv

Multi-view PointNet for 3D Scene Understanding

Maximilian Jaritz, Jiayuan Gu, Hao Su

ICCVW 2019 (Geometry Meets Deep Learning Workshop)

code / arXiv

To What Extent Do Different Neural Networks Learn the Same Representation: A Neuron Activation Subspace Match Approach

Liwei Wang, Lunjia Hu, Jiayuan Gu, Zhiqiang Hu, Yue Wu, Kun He, John Hopcroft

NeurIPS 2019 (Spotlight)

code

Learning Region Features for Object Detection

Jiayuan Gu, Han Hu, Liwei Wang, Yichen Wei, and Jifeng Dai

ECCV 2018

arXiv

Relation Networks for Object Detection

Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei

CVPR 2018 (Oral)

code / arXiv

Services

Conference reviewer: CVPR (2019-2024), ICCV (2019, 2021), ECCV (2020,2024), NeurIPS (2020-2024), ICML (2021,2024), ICLR (2023-2024), AAAI (2021), ICRA (2022), CoRL (2024)

Journal reviewer: PAMI, IJCV, IEEE T-RO