Estimating 3D hand meshes from RGB images robustly is a highly desirable task,made challenging due to the numerous degrees of freedom,and issues such as self-similarity and occlusions. Previous methods generally either use parametric 3D hand models or follow a model-free approach. While the former can be considered more robust,e.g. to occlusions,they are less expressive. We propose a hybrid approach,utilizing a deep neural network and differential rendering based optimization to demonstrably achieve the best of both worlds. In addition,we explore Virtual Reality (VR) as an application. Most VR headsets are nowadays equipped with multiple cameras,which we can leverage by extending our method to the egocentric stereo domain. This extension proves to be more resilient to the above mentioned issues. Finally,as a use-case,we show that the improved image-model alignment can be used to acquire the user's hand texture,which leads to a more realistic virtual hand representation.
|Title of host publication||Proceedings - 2021 International Conference on 3D Vision, 3DV 2021|
|Publisher||Institute of Electrical and Electronics Engineers Inc.|
|Number of pages||10|
|State||Published - 2021|
|Event||9th International Conference on 3D Vision, 3DV 2021 - Virtual, Online, United Kingdom|
Duration: 1 Dec 2021 → 3 Dec 2021
|Name||Proceedings - 2021 International Conference on 3D Vision, 3DV 2021|
|Conference||9th International Conference on 3D Vision, 3DV 2021|
|Period||1/12/21 → 3/12/21|
Bibliographical noteFunding Information:
Acknowledgments. This research was partly supported by Innosuisse funding (Grant No. 34475.1 IP-ICT) and a research grant by FIFA.
© 2021 IEEE.
- 3D reconstruction
- hand shape estimation
- hand tracking
- real time
ASJC Scopus subject areas
- Artificial Intelligence
- Computer Vision and Pattern Recognition