Methods and systems for detecting fraud during biometric identity verification
Described herein are computerized methods and systems for detecting fraud during biometric identity verification. A mobile device captures video comprising a plurality of frames of a person's face. The mobile device extracts from the video frames comprising at least two frames of the person's face from different angles. The mobile device creates a reconstruction of the person's face using the extracted frames and derives signals associated with features of the person's face. The mobile device generates an embedding for each extracted frame using the extracted frames, the three-dimensional reconstruction, and the signals. The mobile device calculates, for each extracted frame, a fraud confidence value based upon the embedding for the extracted frame, attributes of the person's face, and image quality attributes of the extracted image. The mobile device computes a fraud detection decision for the extracted frames based upon the fraud confidence values and the embeddings.
1 . A computer system for detecting fraud during biometric identity verification, the system comprising a mobile computing device coupled to an image capture device and a server computing device, the mobile computing device configured to:
capture, using the image capture device, video comprising a plurality of frames of a person's face;
extract, from the video, at least two frames of the person's face from each of a plurality of different angles;
create a three-dimensional reconstruction of the person's face using the extracted frames;
derive one or more signals associated with features of the person's face using the extracted frames; and
the server computing device configured to:
generate an embedding for each extracted frame using the extracted frames, the three-dimensional reconstruction, and the derived signals;
calculate, for each extracted frame, a fraud confidence value based upon the embedding for the extracted frame, one or more attributes of the person's face, and one or more image quality attributes of the extracted image; and
compute a fraud detection decision for one or more of the extracted frames based upon the fraud confidence values and the embeddings.
2 . The system of claim 1 , wherein, during capture of the video, the mobile computing device generates instructions for movement of the person's head in one or more directions relative to the image capture device.
3 . The system of claim 2 , wherein the instructions comprise a user interface for display to the person, the user interface comprising:
a view of the person's face,
a first visual indicator on a left side of the view of the person's face, and
a second visual indicator on a right side of the view of the person's face,
wherein one or more appearance characteristics of the first visual indicator change as the person's head rotates to the right, and
wherein one or more appearance characteristics of the second visual indicator change as the person's head rotates to the left.
4 . The system of claim 3 , wherein the first visual indicator and the second visual indicator each comprises a line.
5 . The system of claim 4 , wherein:
one or more of a color or a thickness of at least a portion of the line for the first visual indicator changes as the person's head rotates to the right, and
one or more of a color or a thickness of at least a portion of the line for the second visual indicator changes as the person's head rotates to the left.
6 . The system of claim 5 , wherein:
the entire line for the first visual indicator changes color and thickness when the person's head has rotated a minimum number of degrees to the right, and
the entire line for the second visual indicator changes color and thickness when the person's head has rotated a minimum number of degrees to the left.
7 . The system of claim 6 , wherein the mobile computing device generates one or more of an audible alert or a haptic alert when the person's head has rotated a minimum number of degrees to the right and a minimum number of degrees to the left during capture of the video.
8 . The system of claim 6 , wherein the mobile computing device stops capturing the video when the person's head has rotated a minimum number of degrees to the right and a minimum number of degrees to the left.
9 . The system of claim 8 , wherein the mobile computing device determines that the captured video comprises enough frames to enable the server computing device to compute a fraud detection decision prior to stopping the video capture.
10 . The system of claim 8 , wherein the mobile computing device restarts the video capture when the mobile computing device determines that the captured video does not comprise enough frames to enable the server computing device to compute a fraud detection decision.
11 . The system of claim 1 , wherein creating a three-dimensional reconstruction of the person's face using the extracted frames comprises:
generating a plurality of keypoints associated with one or more geometrical locations on the person's face from each extracted frame, wherein the geometrical locations are associated with higher levels of 3D variation;
for each keypoint:
locating the keypoint in each extracted frame, and
determining a relationship between the locations of the keypoint across the extracted frames;
generating a point cloud based upon the keypoints; and
selecting a plurality of camera positions based upon the relationships between the locations of the keypoints across the extracted frames, wherein for each extracted frame, an orientation of the camera is biased toward a face orientation.
12 . The system of claim 11 , wherein the camera positions are selected based upon a minimized reprojection error for the associated keypoints.
13 . The system of claim 11 , wherein generating a plurality of keypoints associated with one or more geometrical locations on the person's face from each extracted frame comprises:
identifying one or more facial landmarks of the person's face, the facial landmarks collectively covering the full face; and
generating one or more keypoints corresponding to each of the facial landmarks.
14 . The system of claim 13 , wherein the mobile computing device removes one or more keypoints that are located outside of the person's face.
15 . The system of claim 1 , wherein deriving one or more signals associated with features of the person's face using the extracted frames comprises:
determining one or more features of the person's face using the extracted frames; and
deriving the one or more signals based upon the determined features of the person's face.
16 . The system of claim 15 , wherein the one or more features of the person's face comprise contour lines of the person's face, regions of the person's face, or both.
17 . The system of claim 1 , wherein generating an embedding for each extracted frame using the extracted frames, the three-dimensional reconstruction, and the derived signals comprises, for each extracted frame:
executing a machine learning classification model using the extracted frame, the three-dimensional reconstruction, and the derived signals as input to generate the embedding.
18 . The system of claim 17 , wherein the machine learning classification model comprises an ensemble of supervised and unsupervised neural networks.
19 . The system of claim 1 , wherein the one or more attributes of the person's face for calculating the confidence value comprise face pose, face size, and face detection confidence.
20 . The system of claim 1 , wherein the one or more image quality attributes of the extracted image comprise image corrections, white balance, brightness, blurriness, and contrast.
21 . The system of claim 1 , wherein the fraud confidence values associated with each extracted frame are aggregated to generate an overall fraud confidence value.
22 . The system of claim 21 , wherein computing a fraud detection decision based upon the fraud confidence values for one or more of the extracted frames comprises:
comparing the overall fraud confidence value to a fraud confidence threshold;
determining that the person in the video is genuine when the overall fraud confidence value falls below the fraud confidence threshold; and
determining that the person in the video is fraudulent when the overall fraud confidence value meets or exceeds the fraud confidence threshold.
23 . The system of claim 1 , wherein the mobile computing device displays the fraud detection decision to the person.
24 . The system of claim 1 , wherein the server computing device transmits the fraud detection decision to a remote computing device for display to a user of the remote computing device.
25 . The system of claim 1 , wherein one or more sensors coupled to the mobile computing device record movement characteristics of the mobile computing device during capture of the video by the image capture device.
26 . The system of claim 25 , wherein the movement characteristics include rotation and acceleration.
27 . The system of claim 25 , wherein the server computing device uses the recorded movement characteristics as input when computing the fraud detection decision.
28 . A computerized method of detecting fraud during biometric identity verification, the method comprising:
capturing, using an image capture device coupled to a mobile computing device, video comprising a plurality of frames of a person's face;
extracting, by the mobile computing device from the video, at least two frames of the person's face from each of a plurality of different angles;
creating, by the mobile computing device, a three-dimensional reconstruction of the person's face using the extracted frames;
deriving, by the mobile computing device, one or more signals associated with features of the person's face using the extracted frames;
generating, by a server computing device, an embedding for each extracted frame using the extracted frames, the three-dimensional reconstruction, and the derived signals;
calculating, by the server computing device for each extracted frame, a fraud confidence value based upon the embedding for the extracted frame, one or more attributes of the person's face, and one or more image quality attributes of the extracted image; and
computing, by the server computing device, a fraud detection decision for one or more of the extracted frames based upon the fraud confidence values and the embeddings.
29 . The method of claim 28 , wherein, during capture of the video, the mobile computing device generates instructions for movement of the person's head in one or more directions relative to the image capture device.
30 . The method of claim 29 , wherein the instructions comprise a user interface for display to the person, the user interface comprising:
a view of the person's face,
a first visual indicator on a left side of the view of the person's face, and
a second visual indicator on a right side of the view of the person's face,
wherein one or more appearance characteristics of the first visual indicator change as the person's head rotates to the right, and
wherein one or more appearance characteristics of the second visual indicator change as the person's head rotates to the left.
31 . The method of claim 30 , wherein the first visual indicator and the second visual indicator each comprises a line.
32 . The method of claim 31 , wherein:
one or more of a color or a thickness of at least a portion of the line for the first visual indicator changes as the person's head rotates to the right, and
one or more of a color or a thickness of at least a portion of the line for the second visual indicator changes as the person's head rotates to the left.
33 . The method of claim 32 , wherein:
the entire line for the first visual indicator changes color and thickness when the person's head has rotated a minimum number of degrees to the right, and
the entire line for the second visual indicator changes color and thickness when the person's head has rotated a minimum number of degrees to the left.
34 . The method of claim 33 , wherein the mobile computing device generates one or more of an audible alert or a haptic alert when the person's head has rotated a minimum number of degrees to the right and a minimum number of degrees to the left during capture of the video.
35 . The system of claim 33 , wherein the mobile computing device stops capturing the video when the person's head has rotated a minimum number of degrees to the right and a minimum number of degrees to the left.
36 . The system of claim 35 , wherein the mobile computing device determines that the captured video comprises enough frames to enable the server computing device to compute a fraud detection decision prior to stopping the video capture.
37 . The system of claim 35 , wherein the mobile computing device restarts the video capture when the mobile computing device determines that the captured video does not comprise enough frames to enable the server computing device to compute a fraud detection decision.
38 . The method of claim 28 , wherein creating a three-dimensional reconstruction of the person's face using the extracted frames comprises:
generating a plurality of keypoints associated with one or more geometrical locations on the person's face from each extracted frame, wherein the geometrical locations are associated with higher levels of 3D variation;
for each keypoint:
locating the keypoint in each extracted frame, and
determining a relationship between the locations of the keypoint across the extracted frames;
generating a point cloud based upon the keypoints; and
selecting a plurality of camera positions based upon the relationships between the locations of the keypoints across the extracted frames, wherein for each extracted frame, an orientation of the camera is biased toward a face orientation.
39 . The method of claim 38 , wherein the camera positions are selected based upon a minimized reprojection error for the associated keypoints.
40 . The method of claim 38 , wherein generating a plurality of keypoints associated with the person's face from each extracted frame comprises:
identifying one or more facial landmarks of the person's face, the facial landmarks collectively covering the full face; and
generating one or more keypoints corresponding to each of the facial landmarks.
41 . The method of claim 38 , wherein the mobile computing device removes one or more keypoints that are located outside of the person's face.
42 . The method of claim 28 , wherein deriving one or more signals associated with features of the person's face using the extracted frames comprises:
determining one or more features of the person's face using the extracted frames; and
deriving the one or more signals based upon the determined features of the person's face.
43 . The method of claim 42 , wherein the one or more features of the person's face comprise contour lines of the person's face, regions of the person's face, or both.
44 . The method of claim 28 , wherein generating an embedding for each extracted frame based upon the extracted frames, the three-dimensional reconstruction, and the derived signals comprises, for each extracted frame:
executing a machine learning classification model using the extracted frame, the three-dimensional reconstruction, and the derived signals as input to generate the embedding.
45 . The method of claim 44 , wherein the machine learning classification model comprises an ensemble of supervised and unsupervised neural networks.
46 . The method of claim 28 , wherein the one or more attributes of the person's face for calculating the confidence value comprise face pose, face size, and face detection confidence.
47 . The method of claim 28 , wherein the one or more image quality attributes of the extracted image comprise image corrections, white balance, brightness, blurriness, and contrast.
48 . The method of claim 28 , wherein the fraud confidence values associated with each extracted frame are aggregated to generate an overall fraud confidence value.
49 . The method of claim 48 , wherein computing a fraud detection decision based upon the fraud confidence values for one or more of the extracted frames comprises:
comparing the overall fraud confidence value to a fraud confidence threshold;
determining that the person in the video is genuine when the overall fraud confidence value falls below the fraud confidence threshold; and
determining that the person in the video is fraudulent when the overall fraud confidence value meets or exceeds the fraud confidence threshold.
50 . The method of claim 28 , wherein the mobile computing device displays the fraud detection decision to the person.
51 . The method of claim 28 , wherein the server computing device transmits the fraud detection decision to a remote computing device for display to a user of the remote computing device.
52 . The method of claim 28 , wherein one or more sensors coupled to the mobile computing device record movement characteristics of the mobile computing device during capture of the video by the image capture device.
53 . The method of claim 52 , wherein the movement characteristics include rotation and acceleration.
54 . The method of claim 52 , wherein the server computing device uses the recorded movement characteristics as input when computing the fraud detection decision.