Embodied AI / 3D Vision / Robot Learning

Jiayuan Gu

I am an Assistant Professor at ShanghaiTech University. My research develops robust robot learning systems in simulation and the real world.

Prior to joining ShanghaiTech, I received my Ph.D. from Computer Science & Engineering at UC San Diego, advised by Prof. Hao Su. I obtained my Bachelor degree at Peking University, advised by Prof. Liwei Wang. I also spent time at Google DeepMind Robotics, Facebook AI, Waymo, Uber ATG, and MSRA.

This website is occasionally updated. For the latest updates (e.g., openings, publications), please check our group website.

Jiayuan Gu

News

  1. 2025.6Our work CAST receives the SIGGRAPH 2025 Best Paper Award.
  2. 2025.5Co-organize the workshop on Human-Robot-Scene at ICCV 2025.
  3. 2024.4I will join ShanghaiTech University as an Assistant Professor.
  4. 2024.1RT-Trajectory is accepted to ICLR 2024 as a spotlight (5%).
  5. 2023.7Co-organize the GMPL workshop at RSS 2023.
  6. 2023.1Two papers are accepted to ICLR 2023.
  7. 2022.8ManiSkill 2022 Challenge is open.
  8. 2022.6I will give a talk at the CVPR 2022 tutorial Building and Working in Environments for Embodied AI.
  9. 2022.4Co-organize the first VOLI workshop at ECCV 2022.
  10. 2020.9Two papers are accepted to NeurIPS 2020.
  11. 2020.7Two papers (spotlight) are accepted to ECCV 2020.
  12. 2019.1Codes for Multi-view PointNet for 3D Scene Understanding are released.

Selected Publications

To see the full list of my research papers, please visit my Google Scholar profile.

CAST teaser

CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image

Kaixin Yao*, Longwen Zhang*, Xinhao Yan, Yan Zeng, Qixuan Zhang, Wei Yang, Lan Xu, Jiayuan Gu, Jingyi Yu

SIGGRAPH 2025 (Best Paper Award)

Project websitearXiv

Point-SAM outdoor example

Point-SAM: Promptable 3D Segmentation Model for Point Clouds

Yuchen Zhou*, Jiayuan Gu*, Tung Yen Chiang, Fanbo Xiang, Hao Su

ICLR 2025

Project websitearXivCode

SimplerEnv teaser

Evaluating Real-World Robot Manipulation Policies in Simulation

Xuanlin Li*, Kyle Hsu*, Jiayuan Gu*, Karl Pertsch†, Oier Mees†, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su‡, Quan Vuong‡, Ted Xiao‡

CoRL 2024

Project websitearXivCode

One-2-3-45++ teaser

One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion

Minghua Liu*, Ruoxi Shi*, Linghao Chen*, Zhuoyang Zhang*, Chao Xu*, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, Hao Su

CVPR 2024

Project website

RT-Trajectory

RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches

Jiayuan Gu, Sean Kirmani, Paul Wohlhart, Yao Lu, Montserrat Gonzalez Arenas, Kanishka Rao, Wenhao Yu, Chuyuan Fu, Keerthana Gopalakrishnan, Zhuo Xu, Priya Sundaresan, Peng Xu, Hao Su, Karol Hausman, Chelsea Finn, Quan Vuong, Ted Xiao

ICLR 2024 (spotlight)

Project websitearXivBlog

RT-Sketch

RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches

Priya Sundaresan, Quan Vuong, Jiayuan Gu, Peng Xu, Ted Xiao, Sean Kirmani, Tianhe Yu, Michael Stark, Ajinkya Jain, Karol Hausman, Dorsa Sadigh*, Jeannette Bohg*, Stefan Schaal*

Project website

ManiSkill2 open-source recipe workflow

An Open-source Recipe for Building Simulated Robot Manipulation Benchmarks

Jiayuan Gu, Linghao Chen, Zhiwei Jia, Fanbo Xiang, Hao Su

ICRA 2023 COMPARE workshop

PDF

ManiSkill2

ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills

Jiayuan Gu, Fanbo Xiang, et.al.

ICLR 2023

Challenge websiteCodePaper

Part reconstruction

Compositionally Generalizable 3D Structure Prediction

Songfang Han, Jiayuan Gu, Kaichun Mo, Li Yi, Siyu Hu, Xuejin Chen, Hao Su

Preprint

CodearXiv

Policy Refactorization

Refactoring Policy for Compositional Generalizability using Self-Supervised Object Proposals

Tongzhou Mu*, Jiayuan Gu*, Zhiwei Jia, Hao Tang, Hao Su

NeurIPS 2020

Project websiteCodearXiv

IterGNN

Towards Scale-Invariant Graph-related Problem Solving by Iterative Homogeneous Graph Neural Networks

Hao Tang, Zhiao Huang, Jiayuan Gu, Bao-Liang Lu, Hao Su

NeurIPS 2020

CodearXiv

Weakly-supervised Shape Completion

Weakly-supervised 3D Shape Completion in the Wild

Jiayuan Gu, Wei-Chiu Ma, Sivabalan Manivasagam, Wenyuan Zeng, Zihao Wang, Yuwen Xiong, Hao Su, Raquel Urtasun

ECCV 2020 (Spotlight)

arXiv

Deep Inverse Solver

Deep Feedback Inverse Problem Solver

Wei-Chiu Ma, Shenlong Wang, Jiayuan Gu, Sivabalan Manivasagam, Antonio Torralba, Raquel Urtasun

ECCV 2020 (Spotlight)

MVPNet

Multi-view PointNet for 3D Scene Understanding

Maximilian Jaritz, Jiayuan Gu, Hao Su

ICCVW 2019 (Geometry Meets Deep Learning Workshop)

CodearXiv

Neural Match

To What Extent Do Different Neural Networks Learn the Same Representation: A Neuron Activation Subspace Match Approach

Liwei Wang, Lunjia Hu, Jiayuan Gu, Zhiqiang Hu, Yue Wu, Kun He, John Hopcroft

NeurIPS 2019 (Spotlight)

Code

Learning Region Features

Learning Region Features for Object Detection

Jiayuan Gu, Han Hu, Liwei Wang, Yichen Wei, and Jifeng Dai

ECCV 2018

arXiv

Relation Networks

Relation Networks for Object Detection

Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei

CVPR 2018 (Oral)

CodearXiv

Services

Conference reviewer: CVPR, ICCV, ECCV, NeurIPS, ICML, ICLR, AAAI, ICRA, CoRL

Journal reviewer: PAMI, IJCV, IEEE T-RO