Facial features tracker with advanced training for natural rendering of human faces in real-time
View Patent ↗Tracking units for facial features with advanced training for natural rendering of human faces in real-time are provided. An example device receives a video stream, and upon detecting a visual face, selects a 3D model from a comprehensive set of head orientation classes. The device determines modifications to the selected 3D model to describe the face, then projects a 2D model of tracking points of facial features based on the 3D model, and controls, actuates, or animates hardware based on the facial features tracking points. The device can switch among an example comprehensive set of 35 different head orientation classes for each video frame, based on suggestions computed from a previous video frame or from yaw and pitch angles of the visual head orientation. Each class of the comprehensive set is trained separately based on a respective collection of automatically marked images for that head orientation class.
1. A method, comprising:
receiving an image of a face from a frame of a video stream;
based on the image, selecting a head orientation class from a comprehensive set of head orientation classes, each head orientation class including a respective 3D model;
determining modifications to the 3D model to describe the face in the image;
projecting a 2D model of tracking points of facial features in an image plane based on the 3D model;
wherein the comprehensive set of head orientation classes comprises 35 different 3D head orientation models representing 35 different head orientations with respect to a camera view of the video stream, wherein each 3D head orientation model has averaged facial features for the respective head orientation of the face with respect to the camera view;
differentiating the 35 different 3D head orientation models of the comprehensive set of head orientation classes by respective yaw angles and pitch angles of each head orientation of each head orientation class; and
controlling, actuating, or animating a piece of hardware based on the tracking points of the facial features.
2. The method of claim 1 , further comprising selecting one of the head orientation classes for the face in a subsequent video frame based on a suggestion computed from a current video frame or from yaw angles and pitch angles associated with one of the head orientation classes.
3. The method of claim 2 , further comprising switching head orientation classes for each adjacent video frame of the video stream during a fast visual movement based on selecting between the suggestion computed from each current video frame and the yaw and pitch angles associated with a head orientation of the face in each subsequent video frame.
4. An apparatus, comprising:
a video input for receiving a video stream;
a face detector for locating an image of a face in a frame of the video stream;
a model generator for selecting a head orientation class based on the image of the face, the head orientation class from a comprehensive set of head orientation classes, each head orientation class including a respective 3D model;
at least a transformation estimator for determining modifications to the 3D model to describe the face in the image;
a facial features projector for creating a 2D model of tracking points of facial features in an image plane based on the 3D model;
wherein the comprehensive set of head orientation classes comprises 35 different 3D head orientation models representing 35 different head orientations with respect to a camera view of the video stream, wherein each 3D head orientation model has averaged facial features for the respective head orientation of the face with respect to the camera view;
differentiating the 35 different 3D head orientation models of the comprehensive set of head orientation classes by respective yaw angles and pitch angles of each head orientation of each head orientation class; and
an export interface for controlling, actuating, or animating a piece of hardware based on the tracking points of the facial features.
5. The apparatus of claim 4 , further comprising a class selector for matching one of the head orientation classes to a face in a subsequent video frame based on a suggestion computed from a current video frame or from yaw angles and pitch angles associated with one of the head orientation classes.
6. The apparatus of claim 5 , wherein the class selector switches head orientation classes for each adjacent video frame of the video stream during a fast visual movement based on selecting between the suggestion computed from each current video frame or the yaw and pitch angles associated with a head orientation of the face in each subsequent video frame.
7. The apparatus of claim 4 , further comprising a model validator to determine a validity or invalidity of the 3D model or of one or more of the tracking points for a given video frame, wherein the model validator signals the validity or the invalidity to an inter-frame face tracker or to the class selector.
8. The method of claim 3 , further comprising accelerating the method in low light levels or under infrared illumination, including accelerating the projecting of the 2D model of tracking points and the controlling, actuating, or animating a piece of hardware, based on selecting the head orientation classes from the comprehensive set of head orientation classes in the low light levels or under the infrared illumination.
9. The apparatus of claim 6 , wherein the class selector switches head orientation classes associated with the comprehensive set of head orientation classes at a low light level or under infrared illumination, wherein the class selector switches the head orientation classes during the fast visual movement based on the apparent yaw and pitch angles associated with a head orientation of the face in each frame of a video stream obtained at a low light level or under infrared illumination to render facial features of the video stream in real-time at a low light level or under infrared illumination.