System and method for off angle three-dimensional face standardization for robust performance
A system uses range and Doppler velocity measurements from a lidar system and images from a video system to estimate a six degree-of-freedom trajectory of a target. The system utilizes a two-stage solution to obtain 3D standardized face representations from non-frontal face views for a statistical learning algorithm. The first stage standardizes the pose (non-frontal 3D face representation) to a frontal view and the second stage uses facial symmetry to fill in missing facial regions due to yaw face pose variations (i.e. rotation about the y-axis).
1. A system for obtaining 3D standardized face representations from non-frontal face views, the system comprising:
a lidar subsystem configured to generate a plurality of three-dimensional (3D) measurements, each of the plurality of 3D measurements corresponding to a point on a target;
a video subsystem comprising a single camera configured to provide a plurality of two-dimensional images of the target; and
a processor configured to:
receive, from the lidar subsystem, the plurality of 3D measurements of the target,
receive, from the video subsystem, the plurality of 2D images of the target,
generate at least one 3D image of the target based on the 3D measurements of the target and the 2D images of the target, wherein the at least one 3D image of the target corresponds to a non-frontal 3D face representation of the target,
transform the non-frontal 3D face representation to a frontal 3D face representation, the frontal 3D face representation having one or more missing regions,
generate a mirror face representation of the frontal 3D face representation, and
selectively fill the one or more missing regions of the frontal 3D face representation with points from the mirror face representation that correspond to the one or more missing regions to generate a 3D standardized face representation, wherein an integrity of the frontal 3D face representation is preserved.
2. The system of claim 1 , wherein the processor configured to transform the non-frontal 3D face representation to the frontal 3D face representation is further configured to:
determine a plurality of candidate face anchor points on the non-frontal 3D face representation;
determine a first anchor point region for each of the plurality of candidate face anchor points;
determine a quality of alignment between the first anchor point region and a reference anchor point region associated with a reference 3D face representation;
determine a face anchor point from the plurality of candidate face anchor points based on the quality of alignment; and
translate the non-frontal 3D face representation to collocate the determined face anchor point with a reference anchor point associated with the reference 3D face representation and transform it to a frontal orientation using an iterative registration process to a frontal face reference model.
3. The system of claim 1 , wherein the one or more missing regions comprises at least one hole in the frontal 3D face representation, and wherein the processor is further configured to:
for each of a plurality of interpolated points in the at least one hole, determine a closest surface point on the mirror face representation; and
merge the determined closest surface point into the frontal 3D face representation.
4. The system of claim 1 , wherein the one or more missing regions comprises at least one eroded face boundary in the frontal 3D face representation, and wherein the processor is further configured to:
trace a right or left boundary of the frontal 3D face representation and a right or left boundary of the mirror face representation;
determine a region that is to be extracted from the mirror face representation and merged with the frontal 3D face representation based on the traced boundaries; and
merge a plurality of points in the determined region of the mirror face representation with the frontal 3D face representation.
5. A method for obtaining a 3D standardized face representation from a non-frontal face view, the method comprising:
receiving, from a lidar system, a plurality of three-dimensional (3D) measurements, each of the plurality of 3D measurements corresponding to a point on a target;
receiving, from a video system comprising a single camera, a plurality of two-dimensional (2D) images of the target;
generating at least one 3D image of the target based on the 3D measurements of the target and the 2D images of the target, wherein the at least one 3D image of the target corresponds to a non-frontal 3D face representation of the target;
transforming the non-frontal 3D face representation of the target to a frontal 3D face representation of the target, the frontal 3D face representation having one or more missing regions;
generating a mirror face representation of the target from the frontal 3D face representation of the target; and
selectively filling the one or more missing regions of the frontal 3D face representation with points from the mirror face representation that correspond to the one or more missing regions to generate a 3D standardized face representation, wherein an integrity of the frontal 3D face representation is preserved.
6. The method of claim 5 , wherein transforming a non-frontal 3D face representation of the target to a frontal 3D face representation of the target further comprises:
determining a plurality of candidate face anchor points on the non-frontal 3D face representation of the target;
determining a first anchor point region for each of the plurality of candidate face anchor points;
determining a quality of alignment between the first anchor point region and a reference anchor point region associated with a reference 3D face representation;
determining a face anchor point from the plurality of candidate face anchor points based on the quality of alignment;
translating the non-frontal 3D face representation of the target to collocate the determined face anchor point with a reference anchor point associated with the reference 3D face representation; and
transforming the non-frontal 3D face representation of the target to a frontal orientation using an iterative registration process to a frontal face reference model.
7. The method of claim 5 , wherein the one or more missing regions comprises at least one hole in the frontal 3D face representation of the target, and wherein the method further comprises:
for each of a plurality of interpolated points in the at least one hole, determining a closest surface point on the mirror face representation of the target; and
merging the determined closest surface point into the frontal 3D face representation of the target.
8. The method of claim 5 , wherein the one or more missing regions comprises at least one eroded face boundary in the frontal 3D face representation of the target, and wherein the method further comprises:
tracing a right or left boundary of the frontal 3D face representation of the target and a right or left boundary of the mirror face representation of the target;
determining a region that is to be extracted from the mirror face representation of the target and to be merged with the frontal 3D face representation of the target based on the traced boundaries; and
merging a plurality of points in the determined region of the mirror face representation of the target with the frontal 3D face representation of the target.
9. A method for obtaining a 3D standardized face representation of a target from a non-frontal face view of the target, the method comprising:
receiving, from a lidar system, a plurality of three-dimensional (3D) measurements, each of the plurality of 3D measurements corresponding to a point on the target;
receiving, from a video system comprising a single camera, a plurality of two-dimensional (2D) images of the target;
generating at least one 3D image of the target based on the 3D measurements of the target and the 2D images of the target, wherein the at least one 3D image of the target corresponds to a non-frontal 3D face representation of the target;
transforming the non-frontal 3D face representation of the target to a frontal 3D face representation of the target, the frontal 3D face representation of the target having one or more missing regions corresponding to portions of the target not in the non-frontal 3D face representation;
generating a mirror face representation of the target from the frontal 3D face representation of the target; and
selectively filling the one or more missing regions of the frontal 3D face representation with points from the mirror face representation that correspond to the one or more missing regions to generate a 3D standardized face representation, wherein an integrity of the frontal 3D face representation is preserved.
10. The system of claim 1 , wherein the processor configured to selectively fill the one or more missing regions of the frontal 3D face representation with points from the mirror face representation that correspond to the one or more missing regions to generate a 3D standardized face representation comprises selectively filling the one or more missing regions of the frontal 3D face representation with points from the mirror face representation without replacing existing regions of the frontal 3D face representation.
11. The method of claim 5 , wherein selectively filling the one or more missing regions of the frontal 3D face representation with points from the mirror face representation that correspond to the one or more missing regions to generate a 3D standardized face representation comprises selectively filling the one or more missing regions of the frontal 3D face representation with points from the mirror face representation without replacing existing regions of the frontal 3D face representation.
12. The method of claim 9 , wherein selectively filling the one or more missing regions of the frontal 3D face representation with points from the mirror face representation that correspond to the one or more missing regions to generate a 3D standardized face representation comprises selectively filling the one or more missing regions of the frontal 3D face representation with points from the mirror face representation without replacing existing regions of the frontal 3D face representation.