RealisticHands: A Hybrid Model for 3D Hand Reconstruction

Michael Seeber, Roi Poranne, Marc Polleyfeys, Martin R. Oswald

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review


Estimating 3D hand meshes from RGB images robustly is a highly desirable task,made challenging due to the numerous degrees of freedom,and issues such as self-similarity and occlusions. Previous methods generally either use parametric 3D hand models or follow a model-free approach. While the former can be considered more robust,e.g. to occlusions,they are less expressive. We propose a hybrid approach,utilizing a deep neural network and differential rendering based optimization to demonstrably achieve the best of both worlds. In addition,we explore Virtual Reality (VR) as an application. Most VR headsets are nowadays equipped with multiple cameras,which we can leverage by extending our method to the egocentric stereo domain. This extension proves to be more resilient to the above mentioned issues. Finally,as a use-case,we show that the improved image-model alignment can be used to acquire the user's hand texture,which leads to a more realistic virtual hand representation.

Original languageEnglish
Title of host publicationProceedings - 2021 International Conference on 3D Vision, 3DV 2021
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages10
ISBN (Electronic)9781665426886
StatePublished - 2021
Event9th International Conference on 3D Vision, 3DV 2021 - Virtual, Online, United Kingdom
Duration: 1 Dec 20213 Dec 2021

Publication series

Name2021 International Conference on 3D Vision (3DV)


Conference9th International Conference on 3D Vision, 3DV 2021
Country/TerritoryUnited Kingdom
CityVirtual, Online

Bibliographical note

Funding Information:
Acknowledgments. This research was partly supported by Innosuisse funding (Grant No. 34475.1 IP-ICT) and a research grant by FIFA.

Publisher Copyright:
© 2021 IEEE.


  • 3D reconstruction
  • hand shape estimation
  • hand tracking
  • monocular
  • real time
  • stereo

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Vision and Pattern Recognition


Dive into the research topics of 'RealisticHands: A Hybrid Model for 3D Hand Reconstruction'. Together they form a unique fingerprint.

Cite this