Methods and systems for authentication of a physical document
Described herein are computerized methods and systems for authentication of a physical document. An image capture device coupled to a mobile device captures a sequence of images of a physical document as at least one of the physical document or the image capture device is rotated, during which the mobile device tracks the physical document throughout the sequence of images, and adjusts operational parameters of the image capture device based upon imaging conditions associated with the physical document. The mobile device selects images from the sequence of images and classifies the physical document using the selected images. The mobile device identifies a region of interest in the physical document using the selected images and the classification. The mobile device reconstructs the region of interest, generates an authentication score for the document using the reconstructed region of interest, and determines whether the physical document is authentic based upon the authentication score.
1 . A system for authenticating a physical document, the system comprising a mobile computing device coupled to an image capture device, the mobile computing device configured to:
capture, using the image capture device, a sequence of images of a physical document in a scene as at least one of the physical document or the image capture device is rotated or tilted along one or more axes, during which the mobile computing device:
tracks the physical document throughout the sequence of images by:
dynamically determining a minimum range of motion for the physical document based upon one or more of the imaging conditions or the operational parameters of the image capture device;
determining whether the rotation or tilt of the physical document or the image capture device satisfies the minimum range of motion; and
instructing a user of the mobile computing device to continue rotating or tilting the physical document or the image capture device until the minimum range of motion is satisfied, and
adjusts one or more operational parameters of the image capture device based upon one or more imaging conditions associated with the physical document, as detected in one or more images of the sequence of images;
select one or more images from the sequence of images and classify the physical document using the selected images;
identify a region of interest in the physical document using the selected images and the classification of the physical document;
reconstruct the region of interest using the selected images;
generate an authentication score for the physical document using the reconstructed region of interest; and
determine whether the physical document is authentic based upon the authentication score.
2 . The system of claim 1 , wherein the minimum range of motion comprises a rotation or tilt of at least a minimum number of degrees in each of one or more planes.
3 . The system of claim 1 , wherein the mobile computing device:
dynamically adjusts one or more lighting parameters of the image capture device during capture of the sequence of images and assesses a signal associated with a region of interest in the physical document; and
instructs the user of the mobile computing device to continue rotating or tilting the physical document or the image capture device until a minimum amount of signal associated with the region of interest is captured and the minimum range of motion is satisfied.
4 . The system of claim 3 , wherein the mobile computing device dynamically adjusts the one or more lighting parameters based upon one or more of: ambient lighting conditions, physical document characteristics, or amount of captured signal associated with the region of interest.
5 . The system of claim 1 , wherein tracking the physical document throughout the sequence of images comprises determining, for each image in the sequence of images, at least one of a location or a six-dimensional pose of the physical document in the image.
6 . The system of claim 1 , wherein the one or more imaging conditions comprise at least one or more of: lighting conditions, focus, or control attributes of the image capture device.
7 . The system of claim 6 , wherein the one or more operational parameters comprise at least one or more of: shutter speed, ISO speed, gain, aperture, flash intensity, flash duration, or light balance.
8 . The system of claim 1 , wherein selecting one or more images from the sequence of images comprises:
determining, for each image in the sequence of images, whether the image is usable or unusable for authentication; and
discarding the image when the image is determined as unusable.
9 . The system of claim 8 , wherein an image is determined to be unusable when: at least a portion of the physical document is occluded or missing, a viewing angle of the physical document exceeds a defined threshold, the image includes noise that exceeds a defined threshold, or at least a portion of the image is blurry.
10 . The system of claim 1 , wherein identifying a region of interest in the physical document using the selected images comprises:
for each image in the selected images:
detecting a location of the physical document in the image;
estimating a pose of the physical document in the image;
cropping a portion of the image based upon the detected location and the pose of the physical document;
estimating one or more characteristics of the physical document based upon the cropped portion of the image; and
aligning the cropped images based upon one or more of the estimated characteristics of the physical document in each cropped image.
11 . The system of claim 10 , wherein the mobile computing device identifies the region of interest in each of the aligned images based upon predefined coordinate values.
12 . The system of claim 1 , wherein the region of interest comprises an optical variable device (OVD).
13 . The system of claim 1 , wherein reconstructing the region of interest using the selected images comprises executing one or more of a robust principal component analysis (PCA) algorithm or a learned alternative mapping on the selected images to reconstruct the region of interest.
14 . The system of claim 1 , wherein the sequence of images of the physical document comprises a plurality of images of a front side of the physical document and a plurality of images of a back side of the physical document.
15 . The system of claim 1 , wherein generating an authentication score for the physical document using the reconstructed region of interest comprises executing one or more machine learning classification models using one or more features of the reconstructed region of interest as input to generate a classification value for the physical document.
16 . The system of claim 15 , wherein the one or more machine learning classification models comprise one or more of: deep learning models, Random Forest algorithms, Support Vector Machines, neural networks, or ensembles thereof.
17 . The system of claim 15 , wherein the classification value comprises at least one of a probability that the physical document is authentic, a confidence score that indicates whether the physical document is authentic, or a similarity metric that indicates whether the physical document is authentic.
18 . The system of claim 15 , wherein at least one of the one or more machine learning classification models is a convolutional neural network.
19 . The system of claim 15 , wherein the one or more machine learning classification models is an ensemble classifier comprised of a plurality of convolutional neural networks.
20 . The system of claim 15 , wherein one or more interpretable methods are used to validate the classification value.
21 . The system of claim 15 , wherein the one or more interpretable methods comprise occlusion of at least a portion of the physical document, perturbation of at least a portion of the physical document, or analysis of a heatmap of at least a portion of the physical document.
22 . The system of claim 21 , wherein an output of the one or more interpretable methods comprises an identification of the reconstructed region of interest that represents proof of the physical document being genuine or fraudulent.
23 . The system of claim 15 , wherein the one or more machine learning classification models are trained using a plurality of genuine documents, a plurality of fraudulent documents, or both.
24 . The system of claim 23 , wherein the classification value generated by the one or more machine learning classification models is a measure of similarity between one or more of the plurality of genuine documents, one or more of the plurality of fraudulent documents, or both.
25 . The system of claim 1 , wherein the mobile computing device preprocesses the sequence of images received from the image capture device prior to selecting the one or more images.
26 . The system of claim 25 , wherein preprocessing the sequence of images comprises one or more of: assessing video quality metrics for the entire sequence of images, detecting a location of the physical document in each image of the sequence of images, and determining one or more quality metrics for each image in the sequence of images.
27 . The system of claim 26 , wherein the video quality metrics comprise a length of the sequence of images, a frames-per-second (FPS) value associated with the sequence of images, and an image resolution associated with the sequence of images.
28 . The system of claim 26 , wherein the one or more quality metrics comprise (i) global image quality metrics including one or more of: glare, blur, white balance, or sensor noise characteristics, (ii) local image quality metrics including one or more of: blur, sharpness, text region confidence, character confidence, or edge detection, or (iii) both the global image quality metrics and the local image quality metrics.
29 . The system of claim 28 , wherein the sensor noise characteristics comprise one or more of: blooming, readout noise, or custom calibration variations.
30 . A computerized method of authenticating a physical document, the method comprising:
capturing, using an image capture device coupled to a mobile computing device, a sequence of images of a physical document in a scene as at least one of the physical document or the image capture device is rotated or tilted along one or more axes, during which the mobile computing device:
tracks the physical document throughout the sequence of images by:
dynamically determining a minimum range of motion for the physical document based upon one or more of the imaging conditions or the operational parameters of the image capture device;
determining whether the rotation or tilt of the physical document or the image capture device satisfies the minimum range of motion; and
instructing a user of the mobile computing device to continue rotating or tilting the physical document or the image capture device until the minimum range of motion is satisfied, and
adjusts one or more operational parameters of the image capture device based upon one or more imaging conditions associated with the physical document, as detected in one or more images of the sequence of images;
selecting, by the mobile computing device, one or more images from the sequence of images and classifying the physical document using the selected images;
identifying, by the mobile computing device, a region of interest in the physical document using the selected images and the classification of the physical document;
reconstructing, by the mobile computing device, the region of interest using the selected images;
generating, by the mobile computing device, an authentication score for the physical document using the reconstructed region of interest; and
determining, by the mobile computing device, whether the physical document is authentic based upon the authentication score.
31 . The method of claim 30 , wherein the minimum range of motion comprises a rotation or tilt of at least a minimum number of degrees in each of one or more planes.
32 . The method of claim 31 , further comprising:
dynamically adjusting one or more lighting parameters of the image capture device during capture of the sequence of images and assesses a signal associated with a region of interest in the physical document; and
instructing the user of the mobile computing device to continue rotating or tilting the physical document or the image capture device until a minimum amount of signal associated with the region of interest is captured and the minimum range of motion is satisfied.
33 . The method of claim 32 , wherein the mobile computing device dynamically adjusts the one or more lighting parameters based upon one or more of: ambient lighting conditions, physical document characteristics, or amount of captured signal associated with the region of interest.
34 . The method of claim 30 , wherein tracking the physical document throughout the sequence of images comprises determining, for each image in the sequence of images, at least one of a location or a six-dimensional pose of the physical document in the image.
35 . The method of claim 30 , wherein the one or more imaging conditions comprise at least one or more of: lighting conditions, focus, or control attributes of the image capture device.
36 . The method of claim 35 , wherein the one or more operational parameters comprise at least one or more of: shutter speed, ISO speed, gain, aperture, flash intensity, flash duration, or light balance.
37 . The method of claim 30 , wherein selecting one or more images from the sequence of images comprises:
determining, for each image in the sequence of images, whether the image is usable or unusable for authentication; and
discarding the image when the image is determined as unusable.
38 . The method of claim 37 , wherein an image is determined to be unusable when: at least a portion of the physical document is occluded or missing, a viewing angle of the physical document exceeds a defined threshold, the image includes noise that exceeds a defined threshold, or at least a portion of the image is blurry.
39 . The method of claim 30 , wherein identifying a region of interest in the physical document using the selected images comprises:
for each image in the selected images:
detecting a location of the physical document in the image;
estimating a pose of the physical document in the image;
cropping a portion of the image based upon the detected location and the pose of the physical document;
estimating one or more characteristics of the physical document based upon the cropped portion of the image; and
aligning the cropped images based upon one or more of the estimated characteristics of the physical document in each cropped image.
40 . The method of claim 39 , wherein the mobile computing device identifies the region of interest in each of the aligned images based upon predefined coordinate values.
41 . The method of claim 30 , wherein the region of interest comprises an optical variable device (OVD).
42 . The method of claim 30 , wherein reconstructing the region of interest using the selected images comprises executing one or more of a robust principal component analysis (PCA) algorithm or a learned alternative mapping on the selected images to reconstruct the region of interest.
43 . The method of claim 30 , wherein the sequence of images of the physical document comprises a plurality of images of a front side of the physical document and a plurality of images of a back side of the physical document.
44 . The method of claim 30 , wherein generating an authentication score for the physical document using the reconstructed region of interest comprises executing one or more machine learning classification models using one or more features of the reconstructed region of interest as input to generate a classification value for the physical document.
45 . The method of claim 44 , wherein the one or more machine learning classification models comprise one or more of: deep learning models, Random Forest algorithms, Support Vector Machines, neural networks, or ensembles thereof.
46 . The method of claim 44 , wherein the classification value comprises at least one of a probability that the physical document is authentic, a confidence score that indicates whether the physical document is authentic, or a similarity metric that indicates whether the physical document is authentic.
47 . The method of claim 44 , wherein at least one of the one or more machine learning classification models is a convolutional neural network.
48 . The method of claim 44 , wherein the one or more machine learning classification models is an ensemble classifier comprised of a plurality of convolutional neural networks.
49 . The method of claim 44 , wherein one or more interpretable methods are used to validate the classification value.
50 . The method of claim 49 , wherein the one or more interpretable methods comprise occlusion of at least a portion of the physical document, perturbation of at least a portion of the physical document, or analysis of a heatmap of at least a portion of the physical document.
51 . The method of claim 50 , wherein an output of the one or more interpretable methods comprises an identification of the reconstructed region of interest that represents proof of the physical document being genuine or fraudulent.
52 . The method of claim 44 , wherein the one or more machine learning classification models are trained using a plurality of genuine documents, a plurality of fraudulent documents, or both.
53 . The method of claim 52 , wherein the classification value generated by the one or more machine learning classification models is a measure of similarity between one or more of the plurality of genuine documents, one or more of the plurality of fraudulent documents, or both.
54 . The method of claim 30 , wherein the mobile computing device preprocesses the sequence of images received from the image capture device prior to selecting the one or more images.
55 . The method of claim 54 , wherein preprocessing the sequence of images comprises one or more of: assessing video quality metrics for the entire sequence of images, detecting a location of the physical document in each image of the sequence of images, and determining one or more quality metrics for each image in the sequence of images.
56 . The method of claim 55 , wherein the video quality metrics comprise a length of the sequence of images, a frames-per-second (FPS) value associated with the sequence of images, and an image resolution associated with the sequence of images.
57 . The method of claim 56 , wherein the one or more quality metrics comprise (i) global image quality metrics including one or more of: glare, blur, white balance, or sensor noise characteristics, (ii) local image quality metrics including one or more of: blur, sharpness, text region confidence, character confidence, or edge detection, or (iii) both the global image quality metrics and the local image quality metrics.
58 . The method of claim 57 , wherein the sensor noise characteristics comprise one or more of: blooming, readout noise, or custom calibration variations.
59 . A system for authenticating a physical document, the system comprising a mobile computing device coupled to an image capture device, the mobile computing device configured to:
capture, using the image capture device, a sequence of images of a physical document in a scene as at least one of the physical document or the image capture device is rotated, during which the mobile computing device:
tracks the physical document throughout the sequence of images, and
adjusts one or more operational parameters of the image capture device based upon one or more imaging conditions associated with the physical document, as detected in one or more images of the sequence of images;
preprocess the sequence of images received from the image capture device by performing one or more of: assessing video quality metrics for the entire sequence of images, detecting a location of the physical document in each image of the sequence of images, and determining one or more quality metrics for each image in the sequence of images, wherein the video quality metrics comprise a length of the sequence of images, a frames-per-second (FPS) value associated with the sequence of images, and an image resolution associated with the sequence of images;
select one or more images from the sequence of images and classify the physical document using the selected images;
identify a region of interest in the physical document using the selected images and the classification of the physical document;
reconstruct the region of interest using the selected images;
generate an authentication score for the physical document using the reconstructed region of interest; and
determine whether the physical document is authentic based upon the authentication score.