Outlier detection in textual data
Systems and methods are disclosed to implement an outlier detection system for text records. In embodiments, the detection system generates a fingerprint for each incoming record so that similar records map to similar fingerprints. Each record is assigned to a closest cluster in a set of clusters based computed distances between on the record's fingerprint and respective cluster fingerprints of the clusters. The cluster fingerprint is dynamically updated to maintain respective a representative fingerprint of its member records. When a new record is received that is not sufficiently close to any cluster, a new cluster is added to the set for the new record. In embodiments, the creation of the new cluster triggers an alert that the new record is a potential outlier. Advantageously, the disclosed detection system can be used to detect outliers in records in near real time, without the need to pre-specify outlier characteristics.
1. A method comprising:
performing, by one or more hardware processors and associated memory that implement a computing system:
receiving a sequence of text records;
generating a record fingerprint for a text record in the sequence;
selecting, from a set of clusters, a closest cluster to the text record based at least in part on distances between the record fingerprint and respective cluster fingerprints of the clusters;
in response to a determination that a distance between the record fingerprint and a cluster fingerprint of the closest cluster exceeds a distance threshold:
creating a new cluster in the set of clusters as an outlier cluster, wherein the outlier cluster comprises the text record as a sole member;
updating a cluster fingerprint of the outlier cluster based at least in part on the record fingerprint of the text record; and
storing information about the outlier cluster as an outlier class in an outlier class library; and
using the outlier class to identify one or more additional outlier records.
2. The method of claim 1 , further comprising performing, by the computing system:
generating a second record fingerprint for a second text record in the sequence;
selecting, from the set of clusters, a second cluster that is closest to the second text record;
in response to a determination that a distance between the second record fingerprint and a cluster fingerprint of the second cluster does not exceed the distance threshold:
assigning the second text record to the second cluster, wherein the assignment updates the cluster fingerprint of the second cluster.
3. The method of claim 1 , further comprising performing, by the computing system:
prior to storing the information about the outlier cluster:
assigning a second text record to the outlier cluster, wherein the assignment updates the cluster fingerprint of the outlier cluster based at least in part on a record fingerprint of the second text record.
4. The method of claim 3 , wherein the assigning of the second text record to the outlier cluster comprises:
incrementing a size of the outlier cluster; and
updating the cluster fingerprint of the outlier cluster based at least in part on the size of the outlier cluster.
5. The method of claim 3 , wherein the assigning of the second text record to the outlier cluster comprises:
updating the cluster fingerprint of the outlier cluster to indicate an average of record fingerprints of members records assigned to the outlier cluster.
6. The method of claim 1 , wherein using the outlier class to identify one or more additional outlier records comprises:
loading the outlier class from the outlier class library into a runtime memory of the computing system.
7. The method of claim 1 , wherein using the outlier class to identify one or more additional outlier records comprises:
performing a search of a log of text records to identify one or more text record whose record fingerprint is within a specified distance of the cluster fingerprint of the outlier class.
8. The method of claim 7 , wherein:
the search is performed in response to user input receive via a graphical user interface (GUI); and
the GUI is configured to display results of the search including the one or more text records identified.
9. The method of claim 1 , wherein the information about the outlier cluster stored in the outlier class includes one or more of:
the cluster fingerprint of the outlier cluster,
an observation time of the text record that caused the creation of the outlier cluster, and
an observed frequency of text records assigned to the outlier cluster.
10. The method of claim 1 , further comprising performing, by the computing system:
updating the outlier class in the outlier class library according to user input received via a graphical user interface (GUI), wherein the user input specifies one or more of:
a name for the outlier class,
a severity level for the outlier class, and
a description of the outlier class.
11. The method of claim 1 , wherein:
the record fingerprint of the text record and the cluster fingerprint of the closest cluster are two bit strings of a same length; and
the distance between the record fingerprint and the cluster fingerprint is a Hamming distance of the two bit strings.
12. The method of claim 1 , further comprising performing, by the computing system:
generating an outlier detection alert in response to the creation of the outlier class.
13. A system comprising:
one or more hardware processors and associated memory that implement a computing system, configured to:
receive a sequence of text records;
generate a record fingerprint for a text record in the sequence;
select, from a set of clusters, a closest cluster to the text record based at least in part on distances between the record fingerprint and respective cluster fingerprints of the clusters;
in response to a determination that a distance between the record fingerprint and a cluster fingerprint of the closest cluster exceeds a distance threshold:
create a new cluster in the set of clusters as an outlier cluster, wherein the outlier cluster comprises the text record as a sole member;
update a cluster fingerprint of the outlier cluster based at least in part on the record fingerprint of the text record; and
store information about the outlier cluster as an outlier class in an outlier class library; and
use the outlier class to identify one or more additional outlier records.
14. The system of claim 13 , wherein the computing system is configured to:
assign a second text record to the outlier cluster, wherein the assignment updates the cluster fingerprint of the outlier cluster based at least in part on a record fingerprint of the second text record.
15. The system of claim 14 , wherein to assign the second text record to the outlier cluster, the computing system is configured to:
increment a size of the outlier cluster; and
update the cluster fingerprint of the outlier cluster based at least in part on the size of the outlier cluster.
16. The system of claim 13 , wherein to use the outlier class to identify one or more additional outlier records, the computing system is configured to:
load the outlier class from the outlier class library into a runtime memory of the computing system.
17. The system of claim 13 , wherein to use the outlier class to identify one or more additional outlier records, the computing system is configured to:
perform a search of a log of text records to identify one or more text record whose record fingerprint is within a specified distance of the cluster fingerprint of the outlier class.
18. The system of claim 13 , wherein the computing system is configured to:
provide a graphical user interface (GUI) configured to receive user input to initiate the search; and
display results of the search via the GUI, the results including the one or more text records identified.
19. The system of claim 13 , wherein the computing system is configured to:
provide a graphical user interface (GUI) configured to receive user input to update the outlier class in the outlier class library, wherein the user input specifies one or more of:
a name for the outlier class,
a severity level for the outlier class, and
a description of the outlier class.
20. One or more non-transitory computer-accessible storage media storing program instructions that when executed on one or more processors of a computing system cause the computing system to:
receive a sequence of text records;
generate a record fingerprint for a text record in the sequence;
select, from a set of clusters, a closest cluster to the text record based at least in part on distances between the record fingerprint and respective cluster fingerprints of the clusters;
in response to a determination that a distance between the record fingerprint and a cluster fingerprint of the closest cluster exceeds a distance threshold:
create a new cluster in the set of clusters as an outlier cluster, wherein the outlier cluster comprises the text record as a sole member;
update a cluster fingerprint of the outlier cluster based at least in part on the record fingerprint of the text record; and
store information about the outlier cluster as an outlier class in an outlier class library; and
use the outlier class to identify one or more additional outlier records.