Jae Shin Yoon (윤재신)

   

I am a Research Scientist at Adobe Research.


I received Ph.D. degree in Computer Science and Engineering at the University of Minnesota (UMN), Twin Cities, advised by Prof. Hyun Soo Park.  I received MS degree in Department of Robotics from Korea Advanced Institute of Science and Technology (KAIST) under the supervision of Prof. In So Kweon. I received my bachelor degree in Electrical Engineering from Hanyang University under the supervision of Prof. Yong Ho Song. I was an exchange student at Kobe University. My general research interest lies in computer&robot vision and machine learning.


Email: jaeyoon [at] adobe.com, jsyoon4325 [at] gmail.com
CV | Google Scholar


Highlight


Project MotionMix (Max Sneak 2022) Motion Dependent Appearance from a Single Camera (CVPR 2022) [Project page] Dynamic Scenes Novel View Synthesis (CVPR 2020)

Publications

Comprehensive Relighting: Generalizable and Consistent Monocular Human Relighting and Harmonization
Junying Wang, Jingyuan Liu, Xin Sun, Krishna Kumar Singh, Zhixin Shu, He Zhang, Jimei Yang, Nanxuan Zhao, Tuanfeng Y. Wang, Simon S. Chen, Ulrich Neumann, Jae Shin Yoon
IEEE Computer Vision and Pattern Recognition (CVPR) 2025
[PDF] [Project Page]
Text2Relight: Creative Portrait Relighting with Text Guidance
Junuk Cha, Mengwei Ren, Krishna Kumar Singh, He Zhang, Yannick Hold-Geoffroy, Seunghyun Yoon, HyunJoon Jung,
{ Jae Shin Yoon*, Seungryul Baek* }
*indicates joint last authors
The Association for the Advancement of Artificial Intelligence (AAAI) 2025
[PDF] [Project Page]
Generative Portrait Shadow Removal
Jae Shin Yoon, Zhixin Shu, Mengwei Ren, Xuaner Zhang, Yannick Hold-Geoffroy, Krishna Kumar Singh, He Zhang
ACM Transactions on Graphics, Vol.43(6) (TOG), SIGGRAPH Asia 2024
[PDF] [Project Page]
High-Fidelity Modeling of Generalizable Wrinkle Deformation
Jingfan Guo, Jae Shin Yoon, Shunsuke Saito, Takaaki Shiratori, Hyun Soo Park
European Conference on Computer Vision (ECCV) 2024
[PDF] [Project Page]
COMPOSE: Comprehensive Portrait Shadow Editing
Andrew Hou, Zhixin Shu, Xuaner Zhang, He Zhang, Yannick Hold-Geoffroy, Jae Shin Yoon, Xiaoming Liu
European Conference on Computer Vision (ECCV) 2024
[PDF] [Project Page]
IntrinsicDiffusion: Joint Intrinsic Layers from Latent Diffusion Models
Jundan Luo, Duygu Ceylan, Jae Shin Yoon, Nanxuan Zhao, Julien Philip, Anna Frühstück, Wenbin Li, Christian Richardt, Tuanfeng Y. Wang
ACM SIGGRAPH 2024
[PDF] [Project Page]
Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction
Junuk Cha, Jihyeon Kim, { Jae Shin Yoon*, Seungryul Baek* }
*indicates joint last authors
IEEE Computer Vision and Pattern Recognition (CVPR) 2024
[PDF] [Project page] [Code]
Relightful Harmonization: Lighting-aware Portrait Background Replacement
Mengwei Ren, Wei Xiong, Jae Shin Yoon, Zhixin Shu, Jianming Zhang, HyunJoon Jung, Guido Gerig, He Zhang
IEEE Computer Vision and Pattern Recognition (CVPR) 2024
[PDF] [Project Page]
Bidirectional Temporal Diffusion Model for Temporally Consistent Human Animation
Tserendorj Adiya, Jae Shin Yoon, Jungeun Lee, Sanghun Kim, Hwasup Lim
International Conference on Learning Representations (ICLR) 2024
[PDF] [Video] [Project page]
3D Reconstruction of Interacting Multi-Person in Clothing from a Single Image
Junuk Cha, Hansol Lee, Jaewon Kim, Bao Truong, { *Jae Shin Yoon, *Seungryul Baek }
*indicates joint last authors
IEEE Winter Conference on Applications of Computer Vision (WACV) 2024
[PDF] [Project Page]
Complete 3D Human Reconstruction from a Single Incomplete Image
Junying Wang, Jae Shin Yoon, Tuanfeng Y. Wang, Krishna Kumar Singh, and Ulrich Neumann
IEEE Computer Vision and Pattern Recognition (CVPR) 2023
[PDF] [Video] [Project page]
Metaverse in the Wild: Modeling, Adapting, and Rendering of 3D Human Avatars from a Single Camera
Jae Shin Yoon
PhD Thesis, Computer Science and Engineering, University of Minnesota 2022
[PDF]
Learning Motion-Dependent Appearance for High-Fidelity Rendering of Dynamic Humans from a Single Camera
Jae Shin Yoon, Duygu Ceylan, Tuanfeng Y. Wang, Jingwan (Cynthia) Lu, Jimei Yang, Zhixin Shu, and Hyun Soo Park
IEEE Computer Vision and Pattern Recognition (CVPR) 2022
[PDF] [Video] [Project page] [Article]
HUMBI: A Large Multiview Dataset of Human Body Expressions and Benchmark Challenge
Jae Shin Yoon, Zhixuan Yu, Jaesik Park, and Hyun Soo Park
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) Journal
[PDF] (Extended version of CVPR 20) [Dataset Page]
Pose-Guided Human Animation from a Single Image in the Wild
Jae Shin Yoon, Lingjie Liu, Vladislav Golyanik, Kripasindhu Sarkar, Hyun Soo Park, and Christian Theobalt
IEEE Computer Vision and Pattern Recognition (CVPR) 2021
[PDF] [Video] [Project page] [Code] [Poster] [5-min video]
Novel View Synthesis of Dynamic Scenes with Globally Coherent Depths from a Monocular Camera
Jae Shin Yoon, Kihwan Kim, Orazio Gallo, Hyun Soo Park, and Jan Kautz
IEEE Computer Vision and Pattern Recognition (CVPR) 2020
[PDF] [PDF_supple] [Video] [Video_supple] [Project Page]
HUMBI: A Large Multiview Dataset of Human Body Expressions
{Zhixuan Yu*, Jae Shin Yoon*}, In Kyu Lee, Prashanth Venkatesh, Jaesik Park, Jihun Yu, and Hyun Soo Park
*indicates joint first authors
IEEE Computer Vision and Pattern Recognition (CVPR) 2020
[PDF] [Video] [Project Page]
Self-Supervised Adaptation of High-Fidelity Face Models for Monocular Performance Tracking
Jae Shin Yoon, Takaaki Shiratori, Shoou-I Yu, and Hyun Soo Park
IEEE Computer Vision and Pattern Recognition (CVPR) 2019 Oral presentation
[PDF] [Supple] [Video] [Project Page] [Poster]
3D Semantic Trajectory Reconstruction from 3D Pixel Continuum
Jae Shin Yoon, Ziwei Li, Hyun Soo Park
IEEE Computer Vision and Pattern Recognition (CVPR) 2018
[PDF] [Video] [Project Page]
KAIST Multi-spectral Day/Night Dataset for Autonomous and Assisted Driving
Yukyung Choi, Namil Kim, Soonmin Hwang, Kibaek Park, Jae Shin Yoon, Kyounghwan An, and In So Kweon
Transactions on Intelligent Transportation Systems (T-ITS)
[PDF] [Project Page]
Pixel-level Matching for Video Object Segmentation using Convolutional Neural Networks
Jae Shin Yoon, Francois Rameau, Junsik Kim, Seokju Lee, Seunghak Shin, and In So Kweon
IEEE International Conference on Computer Vision (ICCV) 2017
[PDF] [Project Page] [Slide] [Poster]
VPGNet: Vanishing Point Guided Network for Lane and Road Marking Detection and Recognition
Seokju Lee, Junsik Km, Jae Shin Yoon, Seunghak Shin, Oleksandr Bailo, Namil Kim, Tae-hee Lee, Hyun Seok Hong, Seung-Hoon Han, In So Kweon
IEEE International Conference on Computer Vision (ICCV) 2017
[PDF] [Project Page]
Robust Road Marking Detection and Recognition Using Density-Based Grouping and Machine Learning Techniques
Oleksandr Bailo, Seokju Lee, Francois Rameau, Jae Shin Yoon, In So Kweon
IEEE Winter Conference on Application and Computer Vision (WACV) 2017
[PDF]
Thermal-Infrared based Drivable Region Detection
Jae Shin Yoon, Kibaek Park, Soonmin Hwang, Namil Kim, Yukyung Choi, Francois Rameau and In So Kweon
IEEE Intelligent Vehicle Symposium (IV) 2016
[PDF] [Project Page] [Dataset]
All-Day Visual Place Recognition: Benchmark Dataset and Baseline
Yukyung Choi, Namil Kim, Kibak Park, Soonmin Hwang, Jae Shin Yoon,In So Kweon
IEEE Computer Vision and Pattern Recognition Workshop 2015 on Visual Place Recognition in Changing Environments (CVPR-VPRICE) 2015
[PDF] [Project Page]
Low-Cost Synchronization for Multispectral Cameras
Yukyung Choi, Namil Kim, Kibak Park, Soonmin Hwang, Jae Shin Yoon, In So Kweon
IEEE International Conference on Ubiquitous Robots and Ambient Intelligence (URAI) 2015
[PDF]
Geometrical calibration of multispectral calibration
Namil Kim, Yukyung Choi, Soonmin Hwang, Kibaek Park, Jae Shin Yoon, In So Kweon
IEEE International Conference on Ubiquitous Robots and Ambient Intelligence
[PDF]
Technical Report
Neural 3D Clothes Retargeting from a Single Image
Jae Shin Yoon, Kihwan Kim, Jan Kautz, and Hyun Soo Park
Technical Report 2021
[PDF]

Dataset

  • Human Multiview Behavioral Imaging Dataset
      •   - Multiview images captured from 107 synchronized cameras
        - 3D gaze, face, hand, body, clothing of 772 subjects
  • Dynamic Scene Multiview Dataset for Depth Estimation and View Synthesis
      •   - 13 Dynamic scenes captured from synchronized 12 multi-views and cell-phone camera
        - Ground-truth for dynamic scene view synthesis and depth estimation
  • Multi-Tasking Recognition for Autonomous Driving Scene
      •   - Ground-truth for classification, detection and segmentation of lanes, road markings, traffic signs, and vanishing point
        - A variety of driving scene images captured from day and night in sunny, cloudy, and rainy days
  • Multi-Sensor Imaging Dataset for Autonomous Driving Day and Night
      •   - Driving scene dataset of the calibrated stereo-vision, thermal camera, velodyne, gps, imu
        - Large-scale day and night multi-sensor dataset
  • Thermal-RGB Road Segmentation Dataset
      •   - Synchronized and aligned thermal-rgb imaging dataset for road segmentation
        - 5941 thermal-rgb image pairs with road mask annotation

    Research Internship

  • Adobe Research, San Jose (Virtual) , USA | 2021 (June - Aug.)
            with Dr. Duygu Ceylan, Dr. Tuanfeng Y. Wang, Dr. Jingwan (Cynthia) Lu, Dr. Jimei Yang, Dr. Zhixin Shu
  • Graphics Vision & Video Lab, Max Planck Institute for Intelligent System, Saarland, Germany | 2020.Aug. - 2021.Jan.
            with Prof. Christian Theobalt, Dr. Lingjie Liu, Dr. Vladislav Golyanik, Dr. Kripasindhu Sarkar
  • NVIDIA Research, Santa Clara, USA | 2019 (Feb. - Aug.)
            with Dr. Kihwan Kim, Dr. Orazio Gallo, Dr. Jan Kautz
  • Facebook Reality Labs, Oculus, Pittsburgh, USA | 2018 (June - Aug.)
            with Dr. Takaaki Shiratori, Dr. Shoou-I Yu
  • Korea Institute of Science and Technology (KIST), Korea | 2017 (Mar. - Aug.)
            with Dr. Hwasup Lim, Dr. Sang Chul Ahn

    • Invited Talk

  • MotionMix: Animating Human from a Single Image
      •   - Adobe Max Sneak [Link] | 2022
  • Metaverse in the Wild: Modeling Adapting and Rendering of 3D Human Avatars from a Single Camera
      •   - Adobe Research (job talk), hosted by Dr. Cynthia Lu | 2022
        - DGIST, AI Colloquium, hosted by Prof. Sunghoon Im | 2022
        - UST, Global Mentoring Conference, hosted by Prof. SangChul Ahn | 2023
  • Novel View Synthesis from Dynamic Scenes
      •   - CVPR Tutorial on Novel View Synthesis: From Depth-Based Warping to Multi-Plane Images and Beyond [Link, Slide] | 2020
        - CVPR Workshop on 3D Scene Understanding for Vision, Graphics, and Robotics (Oral Presentation) [Link] | 2020
  • Self-Supervised Adaptation of High-Fidelity Face Models for Monocular Performance Tracking
      •   - CVPR oral presentation [Video] | 2019
        - Think Tank Team at Samsung Research America in Mountain View, hosted by Dr. Abhijit Bendale | 2019
        - UofM 50th Anniversary Research Showcase (Poster) | 2019
  • 3D Human Behavioral Imaging
      •   - University of Minnesota (UMN) Graduate Research and Discussion Seminars (GRaDS) | 2019
        - Seoul National Univeristy (SNU) Vision Seminar, hosted by Prof. Bohyung Han | 2019
        - Pohang University of Science and Technology (POSTECH) Computer Vision group, hosted by Prof. Minsu Cho | 2019
        - Korea Institute of Science and Technology (KIST), hosted by Dr. Hwasup Lim | 2019
  • 3D Semantic Trajectory Reconstruction from 3D Pixel Continuum
      •   - Amazon Graduate Research Symposium (poster) | 2019
        - University of Minnesota (UMN) VCAI Seminar | 2018
  • Semantics on the Road for All-day Autonomous Driving
      •   - Hyundai Top Talent Forum | 2018

    Honors & Awards

  • Doctoral Consortium at CVPR | 2022
  • Outstanding Reviewer Award, IEEE Conference on Computer Vision and Pattern Recognition | 2021
  • Doctoral Dissertation Fellowship (DDF), University of Minnesota ($ 25,000) | 2021
  • Top Graduate Research Award, UofM Graduate Research and Discussion Seminar (GRaDS) | 2019
  • Summa Cum Laude, Hanyang University (GPA: 4.08/4.5) (94.6 %), Dept. of EE, Hanyang University | 2015
  • Hanyang Brain scholarship, Dept. of EE, Hanyang University ($1,440) | 2014
  • Prize for the top first percentile GPA, Hanyang University ($3,740)
  • Miraeasset Park Hyeon-Joo foundation scholarship for exchange student ($8,565) [Certificate] | 2012
  • Qualcomm Scholarship for outstanding engineer, Qualcomm corp ($3,740) [Certificate] | 2011

    • Patent

  • MODELING SECONDARY MOTION BASED ON THREE-DIMENSIONAL MODELS (P11523-US, 2024)
  • MODIFYING POSES OF TWO-DIMENSIONAL HUMANS IN TWO-DIMENSIONAL IMAGES BY REPOSING THREE-DIMENSIONAL HUMAN MODELS REPRESENTING THE TWO-DIMENSIONAL HUMANS (P11960-US2, 2024)
  • MODIFYING POSES OF TWO-DIMENSIONAL HUMANS IN TWO-DIMENSIONAL IMAGES BY REPOSING THREE-DIMENSIONAL HUMAN MODELS REPRESENTING THE TWO-DIMENSIONAL HUMANS (P11960-CN2, 2024)
  • GENERATING THREE-DIMENSIONAL HUMAN MODELS REPRESENTING TWO-DIMENSIONAL HUMANS IN TWO-DIMENSIONAL IMAGES (P11960-US1, 2024)
  • GENERATING THREE-DIMENSIONAL HUMAN MODELS REPRESENTING TWO-DIMENSIONAL HUMANS IN TWO-DIMENSIONAL IMAGES (P11960-CN1, 2024)
  • View Synthesis for Dynamic Scenes (US Patent 11,546,568, 2023)
  • Vanishing-Guided Lane and Road Marking Detection via Multi-Task Network
      •     Application date: Oct. 31, 2016 (US Provisional Application 52/414,951)