Methods and systems for fraud detection using relative movement of facial features
Described herein are computerized methods and systems for detecting fraud during identity verification. An image capture device of a mobile device captures video comprising a plurality of frames of a person's face and transmits the plurality of frames to a server device. The server detects locations of rigid and non-rigid facial features of the person's face in each of the plurality of frames. The server generates time-series signals based upon a position measurement for the facial features in each of the plurality of frames and extracts classification features from the time-series signals. The server applies a trained machine learning classification model to the extracted classification features to generate a fraud detection decision for the plurality of frames.
1 . A computer system for detecting fraud during identity verification, the system comprising a mobile computing device coupled to an image capture device and a server computing device,
the mobile computing device configured to:
capture, using the image capture device, video comprising a plurality of frames of a person's face; and
transmit the plurality of frames to the server computing device;
the server computing device configured to:
detect a location of one or more facial features of the person's face in each of the plurality of frames, wherein the facial features comprise rigid facial features and non-rigid facial features;
generate a plurality of time-series signals representing the relative movement of the facial features based on the detected location of the facial features in each of the plurality of frames;
extract one or more classification features from the plurality of time-series signals; and
apply a trained machine learning classification model to the extracted classification features to generate a fraud detection decision for the plurality of frames based on whether the classification features are indicative of genuine facial movement.
2 . The system of claim 1 , wherein the mobile computing device transmits the plurality of frames to the server computing device upon determining that a threshold number of frames contain the person's face.
3 . The system of claim 1 , wherein the server computing device applies a facial landmark regression algorithm to each of the plurality of frames to detect the location of the one or more facial features of the person's face.
4 . The system of claim 3 , wherein the non-rigid facial features comprise irises, eyelids, eyebrows, eye corners, and mouth.
5 . The system of claim 4 , wherein the rigid facial features comprise face outline and nose.
6 . The system of claim 1 , wherein generating the plurality of time-series signals representing the relative movement of facial features based on the detected location of the facial features in each of the plurality of frames comprises:
calculating a relative position between the location of each non-rigid facial feature and the location of one or more of the rigid facial features across the plurality of frames; and
generating the plurality of time-series signals based upon the relative positions.
7 . The system of claim 1 , wherein generating the plurality of time-series signals representing the relative movement of facial features based on the detected location of the facial features in each of the plurality of frames comprises:
generating an edge image for each of the plurality of frames;
measuring, in each edge image, a dynamic intensity value of one or more edge pixels associated with the non-rigid facial features;
determining, in each edge image, a static intensity value of one or more edge pixels associated with the rigid facial features; and
generating the plurality of time-series signals based upon a relative intensity value between the dynamic intensity value and the static intensity value across the plurality of edge images.
8 . The system of claim 1 , wherein generating the plurality of time-series signals representing the relative movement of facial features based on the detected location of the facial features in each of the plurality of frames comprises:
measuring an optical flow value for each pixel in each of the plurality of frames;
calculating a dynamic motion value for one or more pixels associated with the non-rigid facial features;
calculating a static motion value for one or more pixels associated with the rigid facial features; and
generating the plurality of time-series signals based upon a relative motion value between the dynamic motion value and the static motion value across the plurality of frames.
9 . The system of claim 1 , wherein extracting one or more classification features from the plurality of time-series signals comprises:
determining one or more statistical attributes of the plurality of time-series signals; and
generating the one or more classification features based upon the statistical attributes.
10 . The system of claim 9 , wherein the one or more statistical attributes comprise one or more moments of the time-series signal distribution, one or more temporal measurements of the time-series signal, and one or more correlation coefficients between the time-series signal and one or more pose measurements of the person's face.
11 . The system of claim 1 , wherein extracting one or more classification features from the plurality of time-series signals comprises:
selecting two or more sequential samples from the time-series signal; and
generating the classification features based upon the two or more sequential samples.
12 . The system of claim 11 , wherein applying the trained machine learning classification model to the extracted classification features to generate the fraud detection decision for the plurality of frames based on whether the classification features are indicative of genuine facial movement comprises:
applying the trained machine learning classification model using the classification features based upon the two or more sequential samples to generate a partial fraud detection score associated with a portion of the video corresponding to the two or more sequential samples;
generating additional classification features based upon additional sequential samples from the time-series signal and applying the trained machine learning classification model using the additional classification features to generate a partial fraud detection score associated each portion of the video corresponding to the additional sequential samples;
calculating an overall fraud detection score for the video based upon the partial fraud detection scores; and
generating the fraud detection decision based upon the overall fraud detection score.
13 . The system of claim 1 , wherein extracting one or more classification features from the plurality of time-series signals comprises:
selecting two or more sequential samples from the time-series signal that correspond to a time of a first user interface event generated during video capture; and
generating the classification features based upon the two or more sequential samples.
14 . The system of claim 13 , wherein applying the trained machine learning classification model to the extracted classification features to generate the fraud detection decision for the plurality of frames based on whether the classification features are indicative of genuine facial movement comprises
applying the trained machine learning classification model using the classification features based upon the two or more sequential samples to generate a partial fraud detection score associated with a portion of the video corresponding to the two or more sequential samples;
generating additional classification features based upon additional sequential samples from the time-series signal that correspond to a time of one or more other user interface events generated during video capture;
applying the trained machine learning classification model using the additional classification features to generate a partial fraud detection score associated each portion of the video corresponding to the additional sequential samples;
calculating an overall fraud detection score for the video based upon the partial fraud detection scores; and
generating the fraud detection decision based upon the overall fraud detection score.
15 . The system of claim 1 , wherein the server computing device trains a machine learning classification model using the extracted classification features as input to create the trained machine learning classification model.
16 . The system of claim 1 , wherein the server computing device transmits the fraud detection decision to the mobile computing device for display.
17 . The system of claim 1 , wherein the server computing device transmits the fraud detection decision to a remote computing device for display to a user of the remote computing device.
18 . The system of claim 1 , wherein the server computing device continuously generates the fraud detection decision for the plurality of frames during video capture.
19 . The system of claim 1 , wherein the server computing device determines that the captured video comprises enough frames to enable the server computing device to compute a fraud detection decision prior to stopping the video capture.
20 . The system of claim 19 , wherein the mobile computing device restarts the video capture when the mobile computing device determines that the captured video does not comprise enough frames to enable the server computing device to compute a fraud detection decision.
21 . The system of claim 1 , wherein the mobile computing device generates one or more visual elements for display on the mobile computing device during video capture.
22 . The system of claim 21 , wherein the one or more visual elements are randomly generated during video capture.
23 . The system of claim 22 , wherein the one or more visual elements include random visual features, random spatial movements, appearances at random times during video capture, disappearances at random times during video capture, appearances at random display locations on the mobile computing device, or disappearances from random display locations on the mobile computing device.
24 . A computerized method of detecting fraud during identity verification, the method comprising:
capturing, using an image capture device of a mobile computing device, video comprising a plurality of frames of a person's face;
transmitting, by the mobile computing device, the plurality of frames to a server computing device;
detecting, by the server computing device, a location of one or more facial features of the person's face in each of the plurality of frames, wherein the facial features comprise rigid facial features and non-rigid facial features;
generating, by the server computing device, a plurality of time-series signals representing the relative movement of the facial features based on the detected location of the facial features in each of the plurality of frames;
extracting, by the server computing device, one or more classification features from the plurality of time-series signals; and
applying, by the server computing device, a trained machine learning classification model to the extracted classification features to generate a fraud detection decision for the plurality of frames based on whether the classification features are indicative of genuine facial movement.
25 . The method of claim 24 , wherein the mobile computing device transmits the plurality of frames to the server computing device upon determining that a threshold number of frames contain the person's face.
26 . The method of claim 24 , wherein the server computing device applies a facial landmark regression algorithm to each of the plurality of frames to detect the location of the one or more facial features of the person's face.
27 . The method of claim 26 , wherein the non-rigid facial features comprise irises, eyelids, eyebrows, eye corners, and mouth.
28 . The method of claim 27 , wherein the rigid facial features comprise face outline and nose.
29 . The method of claim 24 , wherein generating the plurality of time-series signals representing the relative movement of the facial features based on the detected location of the facial features in each of the plurality of frames comprises:
calculating a relative position between the location of each non-rigid facial feature and the location of one or more of the rigid facial features across the plurality of frames; and
generating the plurality of time-series signals based upon the relative positions.
30 . The method of claim 24 , wherein generating the plurality of time-series signals representing the relative movement of the facial features based on the detected location of the facial features in each of the plurality of frames comprises:
generating an edge image for each of the plurality of frames;
measuring, in each edge image, a dynamic intensity value of one or more edge pixels associated with the non-rigid facial features;
determining, in each edge image, a static intensity value of one or more edge pixels associated with the rigid facial features; and
generating the plurality of time-series signals based upon a relative intensity value between the dynamic intensity value and the static intensity value across the plurality of edge images.
31 . The method of claim 24 , wherein generating the plurality of time-series signals representing the relative movement of the facial features based on the detected location of the facial features in each of the plurality of frames comprises:
measuring an optical flow value for each pixel in each of the plurality of frames;
calculating a dynamic motion value for one or more pixels associated with the non-rigid facial features;
calculating a static motion value for one or more pixels associated with the rigid facial features; and
generating the plurality of time-series signals based upon a relative motion value between the dynamic motion value and the static motion value across the plurality of frames.
32 . The method of claim 24 , wherein extracting one or more classification features from the plurality of time-series signals comprises:
determining one or more statistical attributes of the plurality of time-series signals; and
generating the one or more classification features based upon the statistical attributes.
33 . The method of claim 32 , wherein the one or more statistical attributes comprise one or more moments of the time-series signal distribution, one or more temporal measurements of the time-series signal, and one or more correlation coefficients between the time-series signal and one or more pose measurements of the person's face.
34 . The method of claim 24 , wherein extracting one or more classification features from the plurality of time-series signals comprises:
selecting two or more sequential samples from the time-series signal; and
generating the classification features based upon the two or more sequential samples.
35 . The method of claim 34 , wherein applying the trained machine learning classification model to the extracted classification features to generate the fraud detection decision for the plurality of frames based on whether the classification features are indicative of genuine facial movement comprises:
applying the trained machine learning classification model using the classification features based upon the two or more sequential samples to generate a partial fraud detection score associated with a portion of the video corresponding to the two or more sequential samples;
generating additional classification features based upon additional sequential samples from the time-series signal and applying the trained machine learning classification model using the additional classification features to generate a partial fraud detection score associated each portion of the video corresponding to the additional sequential samples;
calculating an overall fraud detection score for the video based upon the partial fraud detection scores; and
generating the fraud detection decision based upon the overall fraud detection score.
36 . The method of claim 24 , wherein extracting one or more classification features from the plurality of time-series signals comprises:
selecting two or more sequential samples from the time-series signal that correspond to a time of a first user interface event generated during video capture; and
generating the classification features based upon the two or more sequential samples.
37 . The method of claim 36 , wherein applying the trained machine learning classification model to the extracted classification features to generate the fraud detection decision for the plurality of frames based on whether the classification features are indicative of genuine facial movement comprises:
applying the trained machine learning classification model using the classification features based upon the two or more sequential samples to generate a partial fraud detection score associated with a portion of the video corresponding to the two or more sequential samples;
generating additional classification features based upon additional sequential samples from the time-series signal that correspond to a time of one or more other user interface events generated during video capture;
applying the trained machine learning classification model using the additional classification features to generate a partial fraud detection score associated each portion of the video corresponding to the additional sequential samples;
calculating an overall fraud detection score for the video based upon the partial fraud detection scores; and
generating the fraud detection decision based upon the overall fraud detection score.
38 . The method of claim 24 , wherein the server computing device trains a machine learning classification model using the extracted classification features as input to create the trained machine learning classification model.
39 . The method of claim 24 , wherein the server computing device transmits the fraud detection decision to the mobile computing device for display.
40 . The method of claim 24 , wherein the server computing device transmits the fraud detection decision to a remote computing device for display to a user of the remote computing device.
41 . The method of claim 24 , wherein the server computing device continuously generates the fraud detection decision for the plurality of frames during video capture.
42 . The method of claim 24 , wherein the server computing device determines that the captured video comprises enough frames to enable the server computing device to compute a fraud detection decision prior to stopping the video capture.
43 . The method of claim 42 , wherein the mobile computing device restarts the video capture when the mobile computing device determines that the captured video does not comprise enough frames to enable the server computing device to compute a fraud detection decision.
44 . The method of claim 24 , wherein the mobile computing device generates one or more visual elements for display on the mobile computing device during video capture.
45 . The method of claim 44 , wherein the one or more visual elements are randomly generated during video capture.
46 . The method of claim 45 , wherein the one or more visual elements include random visual features, random spatial movements, appearances at random times during video capture, disappearances at random times during video capture, appearances at random display locations on the mobile computing device, or disappearances from random display locations on the mobile computing device.