3D Human Reconstruction from A Single Image Using Parametric Model-Conditioned Implicit Representation

E Shanmuga Priya; P Velvizhy; Arul Deepa

doi:10.17485/IJST/v16i44.1618

Article

3D Human Reconstruction from A Single Image Using Parametric Model-Conditioned Implicit Representation

VIEWS 269
PDF 53

Indian Journal of Science and Technology

DOI: 10.17485/IJST/v16i44.1618

Year: 2023, Volume: 16, Issue: 44, Pages: 4054-4062

Original Article

3D Human Reconstruction from A Single Image Using Parametric Model-Conditioned Implicit Representation

E Shanmuga Priya^1*, P Velvizhy¹, Arul Deepa²

¹Department of CSE, College of Engineering, Anna University, Guindy, Chennai, India
²Department of IST, College of Engineering, Anna University, Guindy, Chennai, India

*Corresponding Author
Email: [email protected]

Received Date:30 June 2023, Accepted Date:13 October 2023, Published Date:26 November 2023

This work is licensed under a Creative Commons Attribution 4.0 International License.

Abstract

Background: PaMIR is a novel approach for image-based human reconstruction that utilizes a parametric model-conditioned implicit representation. This method enables the generation of a complete 3D mesh of a human body from a single input image. It uses a neural network that is conditioned on a parametric model of the human body to produce an implicit representation of the 3D surface. Objectives: To develop a novel approach for image based human reconstruction by training neural network and to generate high quality images. Method: In our PaMIR-based reconstruction framework, a novel deep neural network is proposed to regularize the free-form deep implicit function using the semantic features of the parametric model, which improves the generalization ability under the scenarios of challenging poses and various clothing topologies. Findings: The quantitative comparison shows that PaMIR method outperforms the state-of-the-art methods in terms of surface reconstruction accuracy. The errors are also provided when ground-truth SMPL annotations are available to present the upper limit of the reconstruction accuracy if the SMPL estimation is perfect. Overall, this method is more general, more robust and more accurate than HMD, Molding Humans, Deep Human and PIFu. Novelty: A novel depth-ambiguity-aware training loss is further integrated to resolve depth ambiguities and enable successful surface detail reconstruction with imperfect body reference. Finally, we propose a body reference optimization method to improve the parametric model estimation accuracy and to enhance the consistency between the parametric model and the implicit function. With the PaMIR representation, our framework can be easily extended to multi-image input scenarios without the need of multi-camera calibration and pose synchronization.

Keywords: Parametric, framework, NonParametric, HMD, SMPL, Tex2Shape

References

Zheng Z, Yu T, Liu Y, Dai Q. PaMIR: Parametric Model-Conditioned Implicit Representation for Image-Based Human Reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2022;44(6):3170–3184. Available from: https://doi.org/10.1109/TPAMI.2021.3050505
Chen L, Peng S, Zhou X. Towards efficient and photorealistic 3D human reconstruction: A brief survey. Visual Informatics. 2021;5(4):11–19. Available from: https://doi.org/10.1016/j.visinf.2021.10.003
Saxena A, Sun M, Ng AY. Make3d: Learning 3d scene structure from a single still image. IEEE transactions on pattern analysis and machine intelligence. 2021;31:824–840. Available from: https://doi.org/10.1109/TPAMI.2008.132
Priya ES, Velvizhy P, Deepa KA. Neural Style Transfer with distortion handling for Audio and Image. International Conference on Data Science. 2022;p. 1–10. Available from: https://doi.org/10.1109/ICDSAAI55433.2022.10028972
Chen Z, Zhang H. Learning Implicit Fields for Generative Shape Modeling. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2019;p. 5932–5941. Available from: https://doi.org/10.1109/CVPR.2019.00609
Zheng Z, Yu T, Wei Y, Dai Q, Liu Y. DeepHuman: 3D human reconstruction from a single image. Proceedings of the IEEE International Conference on Computer Vision. 2021;p. 7738–7748. Available from: https://doi.org/10.48550/arXiv.1903.06473
Chang Z, Koulieris GA, Shum HPH. 3D Reconstruction of Sculptures from Single Images via Unsupervised Domain Adaptation on Implicit Models. In: Proceedings of the 28th ACM Symposium on Virtual Reality Software and Technology. (Vol. 26, pp. 1-10) ACM. 2022.
Jiang C, Sud A, Makadia A, Huang J, Niebner M, Funkhouser T. Local Implicit Grid Representations for 3D Scenes. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020;p. 6000–6009. Available from: https://doi.org/10.1109/CVPR42600.2020.00604
Guler RA, Kokkinos I. HoloPose: Holistic 3D Human Reconstruction In-The-Wild. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2019;p. 10876–10886. Available from: https://doi.org/10.1109/CVPR.2019.01114
Chibane J, Alldieck T, Pons-Moll G. Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020;p. 6968–6979. Available from: https://doi.org/10.1109/CVPR42600.2020.00700
Gabeur V, Franco JS, Martin X, Schmid C, Rogez G. Moulding humans: Non-parametric 3D human shape estimation from single images. arxiv. 2021;p. 2232–2241. Available from: https://arxiv.org/pdf/1908.00439.pdf
Mescheder L, Oechsle M, Niemeyer M, Nowozin S, Geiger A. Occupancy Networks: Learning 3D Reconstruction in Function Space. Available from: https://doi.org/10.1109/CVPR.2019.00459
Kolotouros N, Pavlakos G, Black M, Daniilidis K. Learning to Reconstruct 3D Human Pose and Shape via Model-Fitting in the Loop. 2019 IEEE/CVF International Conference on Computer Vision (ICCV). 2019;p. 2252–2261. Available from: https://doi.org/10.1109/ICCV.2019.00234
Velvizhy P, Pravi A, Selvi M, Ganapathy S, Kannan A. Fuzzy-based review rating prediction in e-commerce. 2020. Available from: https://doi.org/10.1504/IJBIDM.2020.108034
Gwak J, Christopher B, Choy D, Xu K, Chen S, Savarese. 3D-R2N2: A unified approach for single and multi-view 3D object reconstruction. 2020. Available from: https://doi.org/10.1007/978-3-319-46484-8_38
Deepa KA, Priya S, Velvizhy P. KNN Based Under Sampling: A Cognitive Centred Solution for Imbalanced Dataset Problem in Anaphora Resolution. Available from: https://doi.org/10.17485/IJST/v16i30.1523
Bogo F, Kanazawa A, Lassner C, Gehler P, Romero J, Black MJ. Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image. Computer Vision and Pattern Recognition. 2020. Available from: https://doi.org/10.48550/arXiv.1607.08128
Kocabas M, Athanasiou N, Black MJ. VIBE: Video inference for human body pose and shape estimation. 2020. Available from: https://doi.org/10.48550/arXiv.1912.05656

Copyright

© 2023 Priya et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Published By Indian Society for Education and Environment (iSee)