Apparatus and method for detecting facial pose, image processing system, and storage medium
The present disclosure discloses an apparatus and a method for detecting a facial pose, an image processing system, and a storage medium. The apparatus comprises: an obtaining unit to obtain at least three keypoints of at least one face from an input image based on a pre-generated neural network, wherein coordinates of the keypoints obtained via a layer in the neural network for obtaining coordinates are three-dimensional coordinates; and a determining unit to determine, for the at least one face, a pose of the face based on the obtained keypoints, wherein the determined facial pose includes at least an angle. According to the present disclosure, the accuracy of the three-dimensional coordinates of the facial keypoints can be improved, thus the detection precision of a facial pose can be improved.
1. An apparatus for detecting a facial pose, comprising:
an obtaining unit configured to obtain at least three keypoints of at least one face from an input image based on a neural network, wherein coordinates of the keypoints obtained via a layer in the neural network for obtaining coordinates are three-dimensional coordinates, wherein for the at least one face, the obtaining unit further obtains a pose of the face based on the neural network;
a determining unit configured to determine, for the at least one face, a pose of the face based on the obtained keypoints, wherein the determined facial pose includes at least an angle in one dimension; and
an updating unit configured to update, for the at least one face, the determined facial pose based on the obtained facial pose and weights obtained from the neural network.
2. The apparatus according to claim 1 , wherein in a case where the input image includes faces of different scales, the obtaining unit obtains at least three keypoints of each face simultaneously based on the neural network.
3. The apparatus according to claim 2 , wherein the neural network is pre-generated by:
acquiring a sample image in which faces are labeled with keypoints and acquiring predefined reference regions that can cover the faces in the sample image; wherein the sample image includes at least one face, wherein for a face, at least three keypoints are labeled, and coordinates of the labeled keypoints are three-dimensional coordinates; wherein coordinates of center points of the predefined reference regions are three-dimensional coordinates;
obtaining predicted keypoints of each face for the sample image via the neural network based on the predefined reference regions; and
updating the neural network based on a loss function value, wherein the loss function value is determined based on the labeled keypoints of each face and the predicted keypoints of each face.
4. The apparatus according to claim 1 , wherein the determining unit comprises:
a vector determining unit configured to determine, for the at least one face, a normal vector of the face based on the obtained keypoints; and
a pose estimating unit configured to estimate, for the at least one face, the pose of the face based on the normal vector of the face.
5. The apparatus according to claim 4 , wherein for the at least one face, in a case where the obtaining unit obtains three keypoints of the face, the three keypoints are capable of representing an orientation of the face.
6. The apparatus according to claim 4 , wherein for the at least one face, in a case where the obtaining unit obtains more than three keypoints of the face, the vector determining unit configured to:
select keypoints that are capable of representing an orientation of the face from the obtained keypoints; and
determine the normal vector of the face based on the selected keypoints.
7. The apparatus according to claim 4 , wherein the pose estimating unit configured to:
estimate a yaw angle and a pitch angle of the face respectively based on the normal vector of the face; and
estimate a roll angle of the face based on the normal vector and the estimated yaw angle and pitch angle of the face.
8. The apparatus according to claim 1 , wherein the neural network is pre-generated by:
acquiring a sample image in which faces are labeled with keypoints and poses and acquiring predefined reference regions that can cover the faces in the sample image; wherein the sample image includes at least one face; wherein for a face, at least three keypoints are labeled, and coordinates of the labeled keypoints are three-dimensional coordinates, and the labeled pose includes at least an angle in one dimension; wherein coordinates of center points of the predefined reference regions are three-dimensional coordinates;
obtaining predicted keypoints and predicted pose of each face for the sample image via the neural network based on the predefined reference regions;
updating a portion in the neural network that is used to obtain the predicted keypoints of the face based on a first loss function value and updating a portion in the neural network that is used to obtain the predicted pose of the face based on a second loss function value, wherein the first loss function value is determined based on the labeled keypoints of each face and the predicted keypoints of each face, and the second loss function value is determined based on the labeled pose of each face and the predicted pose of each face.
9. A method for detecting a facial pose comprising:
obtaining at least three keypoints of at least one face from an input image based on a pre-generated neural network, wherein coordinates of the keypoints obtained via a layer in the neural network for obtaining coordinates are three-dimensional coordinates, wherein for the at least one face, the obtaining unit further obtains a pose of the face based on the neural network;
determining, for the at least one face, a pose of the face based on the obtained keypoints, wherein the determined facial pose includes at least an angle in one dimension; and
an updating unit configured to update, for the at least one face, the determined facial pose based on the obtained facial pose and weights obtained from the neural network.
10. The method according to claim 9 , wherein in a case where the input image includes faces of different scales, in the obtaining of keypoints, at least three keypoints of each face are obtained simultaneously based on the neural network.
11. The method according to claim 9 , wherein the determining of the pose of the face comprises:
determining, for the at least one face, a normal vector of the face based on the obtained keypoints; and
estimating, for the at least one face, the pose of the face based on the normal vector of the face.
12. The method according to claim 9 , wherein for the at least one face, in the obtaining of the at least three keypoints, a pose of the face is further obtained based on the neural network.
13. The method according to claim 12 , the method further comprises:
updating, for the at least one face, the determined facial pose based on the obtained facial pose and weights obtained from the neural network.
14. An image processing system comprises:
an image acquisition apparatus for acquiring an image including at least one face;
an apparatus for detecting a facial pose, comprising:
an obtaining unit configured to obtain at least three keypoints of at least one face from the acquired image based on a neural network, wherein coordinates of the keypoints obtained via a layer in the neural network for obtaining coordinates are three-dimensional coordinates, wherein for the at least one face, the obtaining unit further obtains a pose of the face based on the neural network;
a determining unit configured to determine, for the at least one face, a pose of the face based on the obtained keypoints, wherein the determined facial pose includes at least an angle in one dimension;
an updating unit configured to update, for the at least one face, the determined facial pose based on the obtained facial pose and weights obtained from the neural network; and
a post-processing apparatus to perform a corresponding post-processing based on the detected facial pose,
wherein the image acquisition apparatus and the post-processing apparatus are connected to the apparatus for detecting a facial pose via a network.
15. A non-transitory computer-readable storage medium for storing instructions which, when executed by a processor, can cause a facial pose detection to be performed, the instructions comprising:
obtaining at least three keypoints of at least one face from an input image based on a pre-generated neural network; wherein coordinates of the keypoints obtained via a layer in the neural network for obtaining coordinates are three-dimensional coordinates, wherein for the at least one face, a pose of the face based on the neural network is further obtained;
determining, for the at least one face, a pose of the face based on the obtained keypoints; wherein the determined facial pose includes at least an angle in one dimension; and
updating, for the at least one face, the determined facial pose based on the obtained facial pose and weights obtained from the neural network.