Device and method for authenticating user based on facial characteristics and mask characteristics of the user

1. A user authentication device having a processor configured to:

detect a face region in a target object image including at least part of a face of a target object, the target object image being a masked image or an unmasked image,

recognize masked or unmasked in the face region,

extract target object characteristics data from the face region of the target object image, and

call reference data and authenticate if the target object is a registered device user based on the called reference data and the target object characteristics data,

wherein the reference data is generated from an unmasked image of the registered device user,

wherein the called reference data is main reference data or secondary reference data, and when masked is recognized, the main reference data is called, and when unmasked is recognized, the secondary reference data is called,

wherein the secondary reference data is generated using a virtual mask template created based on a user input including mask characteristics,

wherein the main reference data and the secondary reference data are generated from the unmasked image of the registered device user, and

wherein, to generate the secondary reference data, the processor is further configured to:

generate a 3-dimensional (3D) face model of the device user from the face region of the unmasked image of the registered device user and acquire a surface shape of the device user,

generate a virtual mask model, based on the virtual mask template, having a surface shape that matches the surface shape of the device user,

generate a fake masked image of the device user based on the generated virtual mask model and the 3D face model of the device user, and

extract characteristics data from the fake masked image of the device user to generate the secondary reference data.

2. The user authentication device according to claim 1 , wherein, to generate the main reference data, the processor is further configured to:

detect the face region in the unmasked image of the registered device user, and

extract characteristics data from the face region of the unmasked image of the device user to generate the main reference data.

3. The user authentication device according to claim 1 , wherein the user authentication device includes at least one of a feature extractor to extract 2-dimensional (2D) characteristics from an input image or a depth extractor to extract 3D characteristics, and extracts the characteristics data of the reference data and the target object characteristics data using the at least one included extractor.

4. The user authentication device according to claim 3 , wherein the at least one included extractor is a trained neural network to extract the characteristics data from the input image using a training dataset including masked images alone, a training dataset including unmasked images alone, or a training dataset including masked images and unmasked images.

5. The user authentication device according to claim 1 , wherein the mask shape template is generated based on pre-stored mask characteristics, and

the processor is configured to call the mask shape template based on the mask characteristics that matches mask characteristics included in the target object characteristics data extracted from the target object image, when the masked image is inputted as the target object image.

6. The user authentication device according to claim 1 , wherein, to generate the fake masked image, the processor renders the virtual mask model to 2D and projects onto the unmasked image of the device user.

7. The user authentication device according to claim 6 , wherein the processor generates the fake masked image after masked in the target object image is recognized, and

a location of projection in the unmasked image of the device user is a location that matches a mask region in the target object image.

8. The user authentication device according to claim 1 , wherein in case that there is the mask shape template generated by the user input at a time of user authentication, the processor is configured to perform the user authentication only when mask characteristics worn by the target object and the mask characteristics of the mask shape template generated by the user input match.

9. The user authentication device according to claim 1 , wherein the processor is configured to:

record a user recognition result including a previous recognition result and recognized mask characteristics, and

in case that there is a latest mask shape template recorded at a time of user authentication, perform the user authentication only when mask characteristics worn by the target object and mask characteristics of the latest mask shape template match.

10. The user authentication device according to claim 1 , wherein the processor is configured to:

give an authority to use the device to the device user having succeeded the user authentication, and

differently set a range of the authority given when the user authentication succeeded after unmasked in the target object image is recognized and a range of the authority given when the user authentication succeeded after masked in the target object image is recognized.

11. The user authentication device according to claim 1 , wherein the processor is further configured to adjust a mask of the fake masked image to match the mask characteristics of the extracted characteristics data when the unmasked image of the target object is inputted as the target object image.

12. The user authentication device according to claim 11 , wherein the processor adjusts at least one of a size of the mask, a texture of the mask, a brightness of the mask, an illumination level of the mask, a light direction of the mask or a resolution of the mask to match the mask characteristics of the extracted characteristics data.

13. A user authentication method comprising:

detecting a face region in a target object image including at least part of a face of a target object, the target object image being a masked image or an unmasked image,

recognizing masked or unmasked in the face region,

extracting target object characteristics data from the face region of the target object image, and

calling reference data and authenticate if the target object is a registered device user based on the called reference data and the target object characteristics data,

wherein the reference data is generated from an unmasked image of the registered device user,

wherein the called reference data is main reference data or secondary reference data, and when masked is recognized, the main reference data is called, and when unmasked is recognized, the secondary reference data is called,

wherein the secondary reference data is generated using a virtual mask template created based on a user input including mask characteristics including at least one of a mask type, a mask texture, a mask color or a mask shape,

wherein the secondary reference data is generate by:

generating a 3-dimensional (3D) face model of the device user from the face region of the unmasked image of the registered device user and acquire a surface shape of the device user,

generating a virtual mask model, based on the virtual mask template, having a surface shape that matches the surface shape of the device user,

generating a fake masked image of the device user based on the generated virtual mask model and the 3D face model of the device user, and

extracting characteristics data from the fake masked image of the device user to generate the secondary reference data.

14. A user authentication method comprising:

detecting a face region in a target object image including at least part of a face of a target object, the target object image being a masked image or an unmasked image,

recognizing masked or unmasked in the face region,

extracting target object characteristics data from the face region of the target object image,

calling reference data and authenticate if the target object is a registered device user based on the called reference data and the target object characteristics data,

record a user recognition result including a previous recognition result and recognized mask characteristics,

based on a determination that there is a latest mask shape template recorded at a time of user authentication, perform the user authentication only when mask characteristics worn by the target object and mask characteristics of the latest mask shape template match,

wherein the reference data is generated from an unmasked image of the registered device user,

wherein the reference data is generated by:

generating a 3-dimensional (3D) face model of the device user from the face region of the unmasked image of the registered device user and acquire a surface shape of the device user,

generating a virtual mask model, based on a virtual mask template, having a surface shape that matches the surface shape of the device user,

generating a fake masked image of the device user based on the generated virtual mask model and the 3D face model of the device user, and

extracting characteristics data from the fake masked image of the device user to generate the reference data.