Face deformation compensating method for face depth image, imaging device, and storage medium

1 . A face deformation compensating method, comprising:

creating a generic face deformation map by:

obtaining indirect Time-of-Flight (iToF) data of a face sample;

extracting ground truth (GT) data of the face sample;

aligning the GT data with the iToF data, to compute a first depth difference image representing a difference on depth information between the GT data and the iToF data;

detecting and extracting a sample face landmark data of the face sample;

matching the first depth difference image with the sample face landmark data to obtain a second depth difference image, the second depth difference image having the sample face landmark data as marks;

obtaining generic face landmark data; and

based on the generic face landmark data and using the sample face landmark data marked in the second depth difference image, morphing the second depth difference image to create a third depth difference image which is the generic face deformation map and is matched and marked with the generic face landmark data;

obtaining a user face image;

detecting and extracting user face landmark data from the user face image; and

applying the generic face deformation map to the user face image, by:

morphing the generic face deformation map using the generic face landmark data and the user face landmark data; and

compensating depth information in the user face image by using the morphed generic face deformation map so as to correct a face deformation of the user face image.

2 . The method of claim 1 , wherein computing the first depth difference image comprises:

converting point clouds of the iToF data into a mesh;

aligning the GT data with the mesh;

projecting the GT data and the mesh aligned to obtain a projected image of the GT data and a projected image of the iToF data; and

computing the first depth difference image based on the projected images of the GT data and the iToF data.

3 . The method of claim 1 , wherein computing the first depth difference image comprises:

directly projecting the GT data, by using camera intrinsics, to obtain a projected image of the GT data;

projecting point clouds of the iToF data to obtain a projected image of the iToF data; and

computing the first depth difference image based on the projected images of the GT data and the iToF data.

4 . The method of claim 3 , further comprising:

determining whether the projected image of the GT data has been aligned with an image of the face sample.

5 . The method of claim 1 , wherein the applying the generic face deformation map further comprises:

detecting and extracting the user face landmark data from the user face image;

morphing the generic face deformation map to match the generic face landmark data marked in the generic face deformation map with the detected user face landmark data, so as to obtain a fourth depth difference image; and

based on the fourth depth difference image, compensating the depth information of the user face image.

6 . The method of claim 1 , wherein creating the generic face deformation map further comprises:

obtaining capturing parameters of the iToF data of the face sample;

repeating the creating the third depth difference image while changing values of the capturing parameters; and

constructing the generic face deformation map by using a plurality of third depth difference images created based on a plurality of groups of the values of the capturing parameters.

7 . The method of claim 6 , wherein the applying the generic face deformation map further comprises:

obtaining the capturing parameters of the user face image, and interpolating the capturing parameters of the user face image into the generic face deformation map, to obtain a fifth depth difference image marked with the generic face landmark data and corresponding to the capturing parameters of the user face image;

detecting and extracting the user face landmark data from the user face image;

morphing the fifth depth difference image to match the generic face landmark data marked in the fifth depth difference image with the detected user face landmark data, so as to obtain a sixth depth difference image; and

based on the sixth depth difference image, compensating the depth information of the user face image.

8 . The method of claim 6 , wherein the capturing parameters comprise an angle between a gaze direction of a face and camera, and a distance between the camera and the face.

9 . The method of claim 8 , wherein the angle comprises yaw and pitch.

10 . The method of claim 1 , wherein the creating the generic face deformation map further comprises:

obtaining capturing parameters of the iToF data of the face sample;

repeating the creating the third depth difference image while changing values of the capturing parameters;

generating a plurality of the third depth difference images as sample images based on a plurality of groups of the values of the capturing parameters, wherein the face sample is a random face sample;

generating multiple sets of the sample images for a plurality of the face samples, and

computing average values of the multiple sets of the sample images for each group of the values of the capturing parameters, thereby generating the generic face deformation map composed of multiple average third depth difference images.

11 . The method of claim 10 , wherein the applying the generic face deformation map further comprises:

obtaining the capturing parameters of the user face image, and interpolating the capturing parameters of the user face image into the generic face deformation map, to obtain a fifth depth difference image marked with the generic face landmark data and corresponding to the capturing parameters of the user face image;

detecting and extracting the user face landmark data from the user face image;

morphing the fifth depth difference image to match the generic face landmark data marked in the fifth depth difference image with the detected user face landmark data, so as to obtain a sixth depth difference image; and

based on the sixth depth difference image, compensating the depth information of the user face image.

12 . An imaging device, comprising:

a camera to capture a user face image;

a memory which stores a generic face deformation map including a generic face landmark data; and

processing circuitry configured to:

create the generic face deformation map by:

obtaining indirect Time-of-Flight (iToF) data of a face sample;

extracting ground truth (GT) data of the face sample;

aligning the GT data with the iToF data, to compute a first depth difference image representing a difference on depth information between the GT data and the iToF data;

detecting and extracting a sample face landmark data of the face sample;

matching the first depth difference image with the sample face landmark data to obtain a second depth difference image, the second depth difference image having the sample face landmark data as marks;

obtaining generic face landmark data; and

based on the generic face landmark data and using the sample face landmark data marked in the second depth difference image, morphing the second depth difference image to create a third depth difference image which is the generic face deformation map and is matched and marked with the generic face landmark data;

obtain the user face image captured by the camera;

detect and extract user face landmark data from the user face image; and

apply the generic face deformation map to the user face image, by:

morphing the generic face deformation map using the generic face landmark data and the user face landmark data; and

compensating depth information in the user face image by using the morphed generic face deformation map so as to correct a face deformation of the user face image.

13 . The imaging device of claim 12 , wherein the processing circuitry is further configured to:

obtain capturing parameters of the user face image and interpolate the capturing parameters of the user face image into the generic face deformation map, to obtain a depth difference image for compensation marked with the generic face landmark data;

morph the depth difference image for compensation using the generic face landmark data marked in the depth difference image for compensation based on the user face landmark data; and

compensate depth information in the user face image based on the morphed depth difference image for compensation.

14 . A non-transitory storage medium on which a computer readable program is recorded, the program when executed on a processor causes the processor to perform a process comprising:

creating a generic face deformation map by:

obtaining indirect Time-of-Flight (iToF) data of a face sample;

extracting ground truth (GT) data of the face sample;

aligning the GT data with the iToF data, to compute a first depth difference image representing a difference on depth information between the GT data and the iToF data;

detecting and extracting a sample face landmark data of the face sample;

matching the first depth difference image with the sample face landmark data to obtain a second depth difference image, the second depth difference image having the sample face landmark data as marks;

obtaining generic face landmark data; and

based on the generic face landmark data and using the sample face landmark data marked in the second depth difference image, morphing the second depth difference image to create a third depth difference image which is the generic face deformation map and is matched and marked with the generic face landmark data;

obtaining a user face image;

detecting and extracting user face landmark data from the user face image; and

applying the generic face deformation map to the user face image, by:

morphing the generic face deformation map using the generic face landmark data and the user face landmark data; and

compensating depth information in the user face image by using the morphed generic face deformation map so as to correct a face deformation of the user face image.

15 . The non-transitory storage medium of claim 14 , wherein the computing the first depth difference image comprises:

converting point clouds of the iToF data into a mesh;

aligning the GT data with the mesh;

projecting the GT data and the mesh aligned to obtain a projected image of the GT data and a projected image of the iToF data; and

computing the first depth difference image based on the projected images of the GT data and the iToF data.

16 . The non-transitory storage medium of claim 14 , wherein the computing the first depth difference image comprises:

directly projecting the GT data by using camera intrinsics, to obtain a projected image of the GT data;

projecting point clouds of the iToF data to obtain a projected image of the iToF data; and

computing the first depth difference image based on the projected images of the GT data and the iToF data.