Scalable search and ranking of biometric signatures
Systems and methods are directed to detecting fraudulent activity based on biometric signatures. A capture component captures data associated with user interactions with a user interface displayed by a computing device. The captured data includes mouse movement. The captured data is converted into an image of the mouse movement. An image encoder encodes the image into an image vector. Subsequently, an analysis system searches a vector database to identify one or more historical image vectors that are similar to the image vector. The vector database comprises a plurality of image vectors corresponding to previously encoded images, whereby at least some of the plurality of image vectors are flagged as indicating fraudulent activity. A result based on the identified historical image vectors is outputted. The result can comprise a risk score for each of the identified historical image vectors that is based in part on a similarity difference measurement.
1 . A method comprising:
capturing, by a capture component, data associated with user interactions of a user with a user interface displayed by a computing device, the captured data including mouse movement;
generating a graphical image depicting a visual representation of a path of the mouse movement from the captured data;
encoding, by an image encoder, the graphical image into an image vector;
searching, by an analysis system, a vector database comprising a plurality of previously encoded image vectors that are indexed for vector similarity searching, the searching comprising using vector similarity matching to retrieve and rank one or more historical image vectors in the vector database that meet or exceed a similarity threshold in comparison to the image vector, at least some of the plurality of previously encoded image vectors being flagged as indicating fraudulent activity; and
outputting a result based on the one or more identified historical image vectors retrieved from the vector database, the result comprising a risk score for each of the one or more identified historical image vectors that is based in part on a similarity difference measurement.
2 . The method of claim 1 , wherein the captured data further comprises keystrokes or click throughs, the keystrokes or click throughs being included as metadata with the image vector.
3 . The method of claim 1 , wherein:
the captured data further comprises speed of the mouse movement or pressure applied on the computing device during the mouse movement; and
the generating the graphical image comprises generating a graphical image of the mouse movements with the speed or pressure encoded as color.
4 . The method of claim 1 , wherein:
the captured data further comprises captured metadata associated with the user interaction; and
the searching further comprises matching the captured metadata to metadata associated with the one or more historical image vectors, the risk score being adjusted based on the matching of the captured metadata to the metadata associated with the one or more historical image vectors.
5 . The method of claim 4 , wherein the metadata comprises one or more of a location of the user, an Internet Protocol address, a device identifier, a browser specification, a language preference, or a time zone.
6 . The method of claim 1 , wherein the capturing the data comprises capturing a position of a mouse as it moves around the user interface at various times.
7 . The method of claim 1 , wherein:
the capturing the data occurs over multiple browser pages;
the generating the graphical image comprises generating a graphical image depicting a visual representation of a path for each browser page based on data from each browser page;
the searching occurs for each image vector corresponding to the image associated with each browser page; and
the method further comprises combining the one or more historical image vectors for each image vector into a single result for output.
8 . The method of claim 1 , wherein the user interface comprises a web browser, the user interface displaying at least a login page accessed via the web browser.
9 . The method of claim 1 , wherein the user interface comprises an interface displayed by a terminal device, the interface presenting a request for a signature from the user, the data associated with the user interactions being captured during signing of the signature.
10 . The method of claim 1 , wherein the user interface displays a prompt requesting the user to draw a specific object, the data associated with the user interactions being captured during drawing of the specific object.
11 . The method of claim 1 , wherein the searching comprises identifying historical image vectors of other instances of potential fraud that may have occurred in a different account than an account being accessed by the user based on the user interactions.
12 . The method of claim 1 , wherein the encoding is performed by a trained neural network of the image encoder.
13 . The method of claim 1 , further comprising:
determining historical image vectors that have a risk score that transgresses a risk score threshold, wherein the outputting the result is based on the historical image vectors that have the risk score that transgresses the risk score threshold.
14 . The method of claim 1 , wherein the searching is based on cosine similarity.
15 . The method of claim 1 , further comprising:
based on the result, identifying the image vector as indicating fraudulent activity and tagging the image vector as fraudulent; and
based on the image vector being tagged fraudulent, storing the image vector to the vector database for use in future analysis.
16 . A system comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
capturing data associated with user interactions of a user with a user interface displayed by a computing device, the captured data including mouse movement;
generating a graphical image depicting a visual representation of a path of the mouse movement from the captured data;
encoding the graphical image into an image vector;
searching a vector database comprising a plurality of previously encoded image vectors that are indexed for vector similarity searching the searching comprising using vector similarity matching to retrieve and rank one or more historical image vectors in the vector database that meet or exceed a similarity threshold in comparison to the image vector, at least some of the plurality of previously encoded image vectors being flagged as indicating fraudulent activity; and
outputting a result based on the one or more identified historical image vectors retrieved from the vector database, the result comprising a risk score for each of the one or more identified historical image vectors that is based in part on a similarity difference measurement.
17 . The system of claim 16 , wherein:
the captured data further comprises captured metadata associated with the user interaction; and
the searching further comprises matching the captured metadata to metadata associated with the one or more historical image vectors, the risk score being adjusted based on the matching of the captured metadata to the metadata associated with the one or more historical image vectors.
18 . The system of claim 16 , wherein the user interface comprises an interface displayed by a terminal device, the interface presenting a request for a signature from the user, the data associated with the user interactions being captured during signing of the signature.
19 . The system of claim 16 , wherein the user interface displays a prompt requesting the user to draw a specific object, the data associated with the user interactions being captured during drawing of the specific object.
20 . A machine-storage medium comprising instructions which, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
capturing data associated with user interactions of a user with a user interface displayed by a computing device, the captured data including mouse movement;
generating a graphical image depicting a visual representation of a path of the mouse movement from the captured data;
encoding the graphical image into an image vector;
searching a vector database comprising a plurality of previously encoded image vectors that are indexed for vector similarity searching, the searching comprising using vector similarity matching to retrieve and rank one or more historical image vectors in the vector database that meet or exceed a similarity threshold in comparison to the image vector, at least some of the plurality of previously encoded image vectors being flagged as indicating fraudulent activity; and
outputting a result based on the one or more identified historical image vectors retrieved from the vector database, the result comprising a risk score for each of the one or more identified historical image vectors that is based in part on a similarity difference measurement.