EP 4292059 A1 20231220 - MULTIVIEW NEURAL HUMAN PREDICTION USING IMPLICIT DIFFERENTIABLE RENDERER FOR FACIAL EXPRESSION, BODY POSE SHAPE AND CLOTHES PERFORMANCE CAPTURE
Title (en)
MULTIVIEW NEURAL HUMAN PREDICTION USING IMPLICIT DIFFERENTIABLE RENDERER FOR FACIAL EXPRESSION, BODY POSE SHAPE AND CLOTHES PERFORMANCE CAPTURE
Title (de)
NEURONALE MENSCHLICHE MEHRFACHANSICHTSVORHERSAGE MIT IMPLIZIT DIFFERENZIERBAREM RENDERER FÜR GESICHTSAUSDRUCK, KÖRPERHALTUNGSFORM UND KLEIDERLEISTUNGSERFASSUNG
Title (fr)
PRÉDICTION HUMAINE NEURONALE MULTIVUE À L?AIDE D?UN MOTEUR DE RENDU DIFFÉRENTIABLE IMPLICITE POUR L?EXPRESSION FACIALE, LA FORME ET LA POSE DU CORPS, ET LA CAPTURE DE PERFORMANCE DE VÊTEMENTS
Publication
Application
Priority
- US 202163168467 P 20210331
- US 202163279916 P 20211116
- US 202217701991 A 20220323
- IB 2022053034 W 20220331
Abstract (en)
[origin: WO2022208440A1] A neural human performance capture framework (MVS-PERF) captures the skeleton,body shape and clothes displacement, and appearance of a person from a set of calibratedmultiview images. It addresses the ambiguity of predicting the absolute position in monocularhuman mesh recovery, and bridges the volumetric representation from NeRF toanimation-friendly performance capture. MVS-PERF includes three modules to extract featuremaps from multiview images and fuse them to a feature volume, regress the feature volume to anaked human parameters vector, generating an SMPL-X skin-tight body mesh with a skeletalpose, body shape, and expression, and leverage a neural radiance field and a deformation field toinfer the clothes as the displacement on the naked body using differentiable rendering. Clothedbody mesh is obtained by adding the interpolated displacement vectors to the SMPL-X skin-tightbody mesh vertices. The obtained radiance field is used for free-view volumetric rendering of theinput subject.
IPC 8 full level
G06V 40/10 (2022.01); G06T 15/08 (2011.01); G06T 19/20 (2011.01); G06V 10/82 (2022.01)
CPC (source: EP KR)
G06T 7/55 (2016.12 - EP KR); G06T 7/70 (2016.12 - EP KR); G06T 13/40 (2013.01 - EP KR); G06T 15/00 (2013.01 - KR); G06T 17/00 (2013.01 - EP); G06T 17/205 (2013.01 - KR); G06V 10/422 (2022.01 - KR); G06V 10/467 (2022.01 - KR); G06V 10/766 (2022.01 - KR); G06V 10/82 (2022.01 - EP KR); G06V 40/103 (2022.01 - EP KR); G06V 40/174 (2022.01 - KR); G06T 2207/20084 (2013.01 - EP KR); G06T 2207/30196 (2013.01 - EP KR); G06T 2210/12 (2013.01 - KR); G06T 2210/16 (2013.01 - EP KR)
Citation (search report)
See references of WO 2022208440A1
Designated contracting state (EPC)
AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR
Designated extension state (EPC)
BA ME
Designated validation state (EPC)
KH MA MD TN
DOCDB simple family (publication)
WO 2022208440 A1 20221006; CN 116134491 A 20230516; EP 4292059 A1 20231220; JP 2024510230 A 20240306; KR 20230150867 A 20231031
DOCDB simple family (application)
IB 2022053034 W 20220331; CN 202280006134 A 20220331; EP 22715732 A 20220331; JP 2023556536 A 20220331; KR 20237033483 A 20220331