Electronic device and method with face key points detection

1 . An electronic device, comprising:

a memory configured to store instructions; and

a processor configured to execute the instructions to configure the processor to:

obtain a first heat map feature and a first coordinate value feature based on a face image;

generate a first fusion heat map feature by fusing the first heat map feature with a first transformation heat map feature obtained by transforming the first coordinate value feature into a heat map feature type;

generate a first fusion coordinate value feature by fusing the first coordinate value feature with a first transformation coordinate value feature obtained by transforming the first heat map feature into a coordinate value feature type; and

detect a face key point based on the first fusion heat map feature and the first fusion coordinate value feature, the detecting comprising:

obtain a heat map of the face key point based on the first fusion heat map feature;

obtain coordinate value information of the face key point based on the first fusion coordinate value feature; and

detect the face key point by obtaining coordinates of the face key point based on the heat map of the face key point and the coordinate value information of the face key point.

2 . The electronic device of claim 1 , wherein the processor is further configured to obtain the coordinates of the face key point by performing any one of preferential output, an arithmetic mean operation, or a weighted mean operation on coordinates corresponding to the heat map of the face key point and coordinates corresponding to the coordinate value information of the face key point.

3 . The electronic device of claim 1 , wherein the processor is further configured to:

obtain a second heat map feature based on the first fusion heat map feature;

obtain a second coordinate value feature based on the first fusion coordinate value feature;

generate a second fusion heat map feature by fusing the second heat map feature with a second transformation heat map feature obtained by transforming the second coordinate value feature into the heat map feature type;

generate a second fusion coordinate value feature by fusing the second coordinate value feature with a second transformation coordinate value feature obtained by transforming the second heat map feature into the coordinate value feature type; and

detect the face key point based on the second fusion heat map feature and the second fusion coordinate value feature.

4 . The electronic device of claim 3 , wherein the processor is further configured to:

obtain a heat map of the face key point based on the second fusion heat map feature;

obtain coordinate value information of the face key point based on the second fusion coordinate value feature; and

detect the face key point by outputting coordinates of the face key point based on the heat map of the face key point and the coordinate value information of the face key point.

5 . The electronic device of claim 1 , wherein the processor is further configured to:

obtain a first feature comprising information on a generic feature of the face based on the face image;

obtain a second feature associated with the first heat map feature and a third feature associated with the first coordinate value feature based on the first feature;

obtain the first heat map feature based on the second feature; and

obtain the first coordinate value feature based on the third feature.

6 . The electronic device of claim 5 , wherein the processor is further configured to:

perform any one of resolution segmentation, channel segmentation, or duplication on the first feature; and

obtain the second feature and the third feature based on a result of the performing.

7 . The electronic device of claim 1 , wherein the processor is further configured to detect the face key point based on a lightweight neural network-based heat map regression model and a lightweight neural network-based coordinate value regression model.

8 . The electronic device of claim 1 , wherein the processor is further configured to:

generate the first fusion heat map feature by performing element level multiplication, element level addition, channel splicing, or pixel splicing on the first transformation heat map feature and the first heat map feature; and

generate the first fusion coordinate value feature by performing another element level multiplication, element level addition, channel splicing, or pixel splicing on the first transformation coordinate value feature and the first coordinate value feature.

9 . An operating method of an electronic device, the operating method comprising:

obtaining, based on a face image, a first heat map feature and a first coordinate value feature;

generating a first fusion heat map feature by fusing the first heat map feature with a first transformation heat map feature obtained by transforming the first coordinate value feature into a heat map feature type;

generating a first fusion coordinate value feature by fusing the first coordinate value feature with a first transformation coordinate value feature obtained by transforming the first heat map feature into a coordinate value feature type; and

detecting, based on the first fusion heat map feature and the first fusion coordinate value feature, a face key point, wherein the detecting comprises:

obtaining, based on the first fusion heat map feature, a heat map of the face key point;

obtaining, based on the first fusion coordinate value feature, coordinate value information of the face key point; and

detecting, based on the heat map of the face key point and the coordinate value information of the face key point, the face key point by obtaining coordinates of the face key point.

10 . The operating method of claim 9 , wherein the detecting of the face key point by obtaining the coordinates of the face key point comprises obtaining the coordinates of the face key point by performing any one of preferential output, an arithmetic mean operation, or a weighted mean operation on coordinates corresponding to the heat map of the face key point and coordinates corresponding to the coordinate value information of the face key point.

11 . The operating method of claim 9 , wherein the detecting, based on the first fusion heat map feature and the first fusion coordinate value feature, of the face key point comprises:

obtaining, based on the first fusion heat map feature, a second heat map feature;

obtaining, based on the first fusion coordinate value feature, a second coordinate value feature;

generating a second fusion heat map feature by fusing the second heat map feature with a second transformation heat map feature obtained by transforming the second coordinate value feature into the heat map feature type;

generating a second fusion coordinate value feature by fusing the second coordinate value feature with a second transformation coordinate value feature obtained by transforming the second heat map feature into the coordinate value feature type; and

detecting the face key point based on the second fusion heat map feature and the second fusion coordinate value feature.

12 . The operating method of claim 11 , wherein the detecting, based on the second fusion heat map feature and the second fusion coordinate value feature, of the face key point comprises:

obtaining, based on the second fusion heat map feature, a heat map of the face key point;

obtaining, based on the second fusion coordinate value feature, coordinate value information of the face key point; and

detecting, based on the heat map of the face key point and the coordinate value information of the face key point, the face key point by outputting coordinates of the face key point.

13 . The operating method of claim 9 , wherein the obtaining of the first heat map feature and the first coordinate value feature comprises:

obtaining, based on the face image, a first feature comprising information on a generic feature of the face;

obtaining, based on the first feature, a second feature associated with the first heat map feature and a third feature associated with the first coordinate value feature;

obtaining, based on the second feature, the first heat map feature; and

obtaining, based on the third feature, the first coordinate value feature.

14 . The operating method of claim 13 , wherein the obtaining, based on the first feature, of the second feature associated with the first heat map feature and the third feature associated with the first coordinate value feature comprises:

performing any one of resolution segmentation, channel segmentation, or duplication on the first feature; and

obtaining, based on a result of the performing, the second feature and the third feature.

15 . The operating method of claim 9 , wherein the face key point is detected based on a lightweight neural network-based heat map regression model and a lightweight neural network-based coordinate value regression model.

16 . The operating method of claim 9 , wherein

the generating of the first fusion heat map feature comprises generating the first fusion heat map feature by performing element level multiplication, element level addition, channel splicing, or pixel splicing on the first transformation heat map feature and the first heat map feature, and

the generating of the first fusion coordinate value feature comprises generating the first fusion coordinate value feature by performing element level multiplication, element level addition, channel splicing, or pixel splicing on the first transformation coordinate value feature and the first coordinate value feature.