Jiayuan Gu

I recently got my Ph.D. in Computer Science & Engineering (CSE) at UC San Diego, advised by Prof. Hao Su. Prior to UCSD, I received my Bachelor degree at the Peking University, advised by Prof. Liwei Wang.

My research topics include Embodied AI and 3D vision. Specifically, I am interested in developing robust robot learning systems in both simulation and the real world.

Starting in Summer 2024, I will join ShanghaiTech University as an Assistant Professor at the School of Information Science and Technology. If you are interested in working with me, please send me your CV.

Email  /  Google Scholar  /  Github

profile photo
News

New [2024.4] I will join ShanghaiTech University as an Assistant Professor.

[2024.1] RT-Trajectory is accepted to ICLR 2024 as a spotlight (5%).

[2023.7] I co-organize the GMPL workshop at RSS 2023.

[2023.1] Two papers are accepted to ICLR 2023.

[2022.8] ManiSkill 2022 Challenge is open!

[2022.6] I give a talk at the CVPR 2022 tutorial Building and Working in Environments for Embodied AI.

[2022.4] I co-organize the first VOLI workshop at ECCV 2022.

[2020.9] Two papers are accepted to NeurIPS 2020.

[2020.7] Two papers (spotlight) are accepted to ECCV 2020.

[2019.01.12] Codes for Multi-view PointNet for 3D Scene Understanding are released.

Research
simpler-env
Point-SAM: Promptable 3D Segmentation Model for Point Clouds
Yuchen Zhou*, Jiayuan Gu*, Tung Yen Chiang, Fanbo Xiang, Hao Su
project website / arXiv / Code
simpler-env
Evaluating Real-World Robot Manipulation Policies in Simulation
Xuanlin Li*, Kyle Hsu*, Jiayuan Gu*, Karl Pertsch†, Oier Mees†, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su‡, Quan Vuong‡, Ted Xiao‡
RSS 2024 Data Generation for Robotics Workshop (spotlight)
project website / arXiv / Code
One-2-3-45++
One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion
Minghua Liu*, Ruoxi Shi*, Linghao Chen*, Zhuoyang Zhang*, Chao Xu*, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, Hao Su
CVPR 2024
project website
RT-Trajectory
RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches
Jiayuan Gu, Sean Kirmani, Paul Wohlhart, Yao Lu, Montserrat Gonzalez Arenas, Kanishka Rao, Wenhao Yu, Chuyuan Fu, Keerthana Gopalakrishnan, Zhuo Xu, Priya Sundaresan, Peng Xu, Hao Su, Karol Hausman, Chelsea Finn, Quan Vuong, Ted Xiao
ICLR 2024 (spotlight)
project website / arXiv / blog
RT-Sketch
RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches
Priya Sundaresan, Quan Vuong, Jiayuan Gu, Peng Xu, Ted Xiao, Sean Kirmani, Tianhe Yu, Michael Stark, Ajinkya Jain, Karol Hausman, Dorsa Sadigh*, Jeannette Bohg*, Stefan Schaal*
project website
ManiSkill2 Open-source Recipe
ManiSkill2
Rearrangement
Multi-skill Mobile Manipulation for Object Rearrangement
Jiayuan Gu, Devendra Singh Chaplot, Hao Su, Jitendra Malik
ICLR 2023 (spotlight)
project website / code / arxiv
Rank 1st in Habitat Rearrangement Challenge 2022
Part Reconstruction
Compositionally Generalizable 3D Structure Prediction
Songfang Han, Jiayuan Gu, Kaichun Mo, Li Yi, Siyu Hu, Xuejin Chen, Hao Su
Preprint
code / arXiv
Policy Refactorization
IterGNN
Weakly-supervised Shape Completion
Weakly-supervised 3D Shape Completion in the Wild
Jiayuan Gu, Wei-Chiu Ma, Sivabalan Manivasagam, Wenyuan Zeng, Zihao Wang, Yuwen Xiong, Hao Su, Raquel Urtasun
ECCV 2020 (Spotlight)
arXiv
Deep Inverse Solver
Deep Feedback Inverse Problem Solver
Wei-Chiu Ma, Shenlong Wang, Jiayuan Gu, Sivabalan Manivasagam, Antonio Torralba, Raquel Urtasun
ECCV 2020 (Spotlight)
arXiv
MVPNet
Multi-view PointNet for 3D Scene Understanding
Maximilian Jaritz, Jiayuan Gu, Hao Su
ICCVW 2019 (Geometry Meets Deep Learning Workshop)
code / arXiv
Neural Match
To What Extent Do Different Neural Networks Learn the Same Representation: A Neuron Activation Subspace Match Approach
Liwei Wang, Lunjia Hu, Jiayuan Gu, Zhiqiang Hu, Yue Wu, Kun He, John Hopcroft
NeurIPS 2019 (Spotlight)
code
Learning Region Features
Learning Region Features for Object Detection
Jiayuan Gu, Han Hu, Liwei Wang, Yichen Wei, and Jifeng Dai
ECCV 2018
arXiv
Relation Networks
Relation Networks for Object Detection
Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei
CVPR 2018 (Oral)
code / arXiv
Services

Conference reviewer: CVPR (2019-2024), ICCV (2019, 2021), ECCV (2020,2024), NeurIPS (2020-2024), ICML (2021,2024), ICLR (2023-2024), AAAI (2021), ICRA (2022), CoRL (2024)

Journal reviewer: PAMI, IJCV, IEEE T-RO