High-speed anomaly detection for large datasets
The various implementations provide for techniques to ingest raw data into a set of events and sample a subset of the events for initial processing via high-speed anomaly detection. The high-speed anomaly detection techniques use a first set of sensitivities that enable fast detection of potential anomalies within the subset of events. The subset of events is also included in a full set of events that are stored in an index and processed via full-scale anomaly detection. The full-scale anomaly detection uses a second set of sensitivities that enable the system to detect anomalies from the full set of events accurately. Upon detecting an anomaly using the high-speed or full-scale detection techniques, the system performs actions to notify entities or automatically mitigate the effects of the anomaly such as mitigation actions to quell potentially fraudulent activity.
1 . A computer-implemented method, comprising:
generating, by an edge processor, a set of events based on data received from a data source;
sampling, from the set of events, a subset of events, wherein the subset of events is selected by the edge processor based on a sample flag value associated with each event of the set of events;
identifying, via a set of high speed anomaly detection operations processing the subset of events using a first set of sensitivity thresholds, a first set of anomalies associated with the subset of events wherein the first set of sensitivity thresholds corresponds to a first quantity of events that match predefined search criteria;
identifying, via a set of full scale anomaly detection operations processing the set of events using a second set of sensitivity thresholds that correspond to thresholds of a greater quantity than the first set of sensitivity thresholds, a second set of anomalies associated with the set of events, wherein the first set of anomalies comprises a subset of the second set of anomalies, and wherein the first set of anomalies are identified prior to the second set of anomalies, wherein at least one of the first set of anomalies or the second set of anomalies includes identifying an event or series of events that deviate from a standard, user accounts with anomalous metadata, anomalous combinations of transactions, or a high rate of errors from a particular internet protocol (IP) address;
forwarding, by the edge processor, a reduced dataset relative to the data received from the data source to a data intake and query system, wherein the reduced dataset includes the first set of anomalies and the second set of anomalies;
generating a user interface that includes an indication of a likelihood of a presence of either the first set of anomalies or the second set of anomalies, wherein the user interface is configured for display on a computing device, and
blocking an identified internet protocol (IP) address associated with either the first set of anomalies or the second set of anomalies.
2 . The computer-implemented method of claim 1 , further comprising: for each event in the set of events, adding a sampling field value,
wherein sampling the subset of events comprises identifying each event in the set of events having a sampling field value that meets sampling criteria.
3 . The computer-implemented method of claim 2 , wherein:
sampling the subset of event comprises executing an object query specifying the sampling criteria, and
the object query is associated with a data model representing a view of the set of events.
4 . The computer-implemented method of claim 1 , wherein: an edge processor receives the data from the data source, the edge processor executes at least a portion of the first set of anomaly detection operations, and the edge processor executes at least a portion of the second set of anomaly detections operations.
5 . The computer-implemented method of claim 1 , further comprising:
indexing the subset of events into a first index of a data store, wherein the first set of anomaly detection operations is performed on the subset of events in the first index; and
indexing the set of events in a second index of the data store, wherein the second set of anomaly detection operations is performed on the subset of events in the second index.
6 . The computer-implemented method of claim 1 , further comprising:
upon identifying the first set of anomalies or the second set of anomalies, generating a notification report.
7 . The computer-implemented method of claim 6 , wherein the notification report includes a label for each of the first set of anomalies.
8 . The computer-implemented method of claim 1 , wherein:
the first set of anomaly detection operations processing the subset of events comprises a trained machine learning (ML) model receiving the subset of events as inputs,
the trained ML model generates a sensitivity threshold from a training set of data, and the trained ML model analyzes the subset of events for an anomaly compared to the sensitivity threshold.
9 . The computer-implemented method of claim 1 , wherein identifying the first set of anomalies comprises:
detecting a number of events in the subset of events that meets a first search criterion; and identifying a first anomaly included in the first set of anomalies upon determining that the number of events exceeds a sensitivity threshold associated with the first anomaly.
10 . The computer-implemented method of claim 1 , wherein:
each event in the set of events includes a portion of unstructured raw machine data reflecting activity in an information technology environment, and
the first set of anomalies or the second set of anomalies indicate a likelihood of fraudulent activity associated with the information technology environment.
11 . The computer-implemented method of claim 1 , wherein:
the first set of anomaly detection operations uses a first set of sensitivity thresholds on the subset of events to identify the first set of anomalies;
the second set of anomaly detection operations uses a second set of sensitivity thresholds on the set of events to identify the second set of anomalies, and
the second set of sensitivity thresholds is greater than the first set of sensitivity thresholds.
12 . The computer-implemented method of claim 1 , wherein the subset of events comprises 10% or fewer of the set of events.
13 . A computing device, comprising: a processor; and
a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations including:
generating, by an edge processor, a set of events based on data received from a data source;
sampling, from the set of events, a subset of events, wherein the subset of events is selected by the edge processor based on a sample flag value associated with each event of the set of events;
identifying, via a set of high speed anomaly detection operations processing the subset of events using a first set of sensitivity thresholds, a first set of anomalies associated with the subset of events wherein the first set of sensitivity thresholds corresponds to a first quantity of events that match predefined search criteria;
identifying, via a set of full scale anomaly detection operations processing the set of events using a second set of sensitivity thresholds that correspond to thresholds of a greater quantity than the first set of sensitivity thresholds, a second set of anomalies associated with the set of events, wherein the first set of anomalies comprises a subset of the second set of anomalies, and wherein the first set of anomalies are identified prior to the second set of anomalies, wherein at least one of the first set of anomalies or the second set of anomalies includes identifying an event or series of events that deviate from a standard, user accounts with anomalous metadata, anomalous combinations of transactions, or a high rate of errors from a particular internet protocol (IP) address;
forwarding, by the edge processor, a reduced dataset relative to the data received from the data source to a data intake and query system, wherein the reduced dataset includes the first set of anomalies and the second set of anomalies;
generating a user interface that includes an indication of a likelihood of a presence of either the first set of anomalies or the second set of anomalies, wherein the user interface is configured for display on a computing device, and
blocking an identified internet protocol (IP) address associated with either the first set of anomalies or the second set of anomalies.
14 . The computer device of claim 13 , wherein:
the operations further include, for each event in the set of events, adding a sampling field value,
sampling the subset of events comprises executing an object query specifying sampling criteria, where each event in the set of events has a sampling field value that meets the sampling criteria, and
the object query is associated with a data model representing a view of the set of events.
15 . The computer device of claim 13 , wherein:
an edge processor receives the data from the data source,
the edge processor executes at least a portion of the first set of anomaly detection operations, and
the edge processor executes at least a portion of the second set of anomaly detections operations.
16 . The computer device of claim 13 , wherein further the operations further include:
indexing the subset of events into a first index of a data store, wherein the first set of anomaly detection operations is performed on the subset of events in the first index; and
indexing the set of events in a second index of the data store, wherein the second set of anomaly detection operations is performed on the subset of events in the second index.
17 . The computer device of claim 13 , wherein the first set of anomalies and the second set of anomalies each includes one or more anomalies.
18 . One or more non-transitory computer-readable media having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform operations including:
generating, by an edge processor, a set of events based on data received from a data source;
sampling, from the set of events, a subset of events, wherein the subset of events is selected by the edge processor based on a sample flag value associated with each event of the set of events;
identifying, via a set of high speed anomaly detection operations processing the subset of events using a first set of sensitivity thresholds, a first set of anomalies associated with the subset of events wherein the first set of sensitivity thresholds corresponds to a first quantity of events that match predefined search criteria;
identifying, via a set of full scale anomaly detection operations processing the set of events using a second set of sensitivity thresholds that correspond to thresholds of a greater quantity than the first set of sensitivity thresholds, a second set of anomalies associated with the set of events, wherein the first set of anomalies comprises a subset of the second set of anomalies, and wherein the first set of anomalies are identified prior to the second set of anomalies, wherein at least one of the first set of anomalies or the second set of anomalies includes identifying an event or series of events that deviate from a standard, user accounts with anomalous metadata, anomalous combinations of transactions, or a high rate of errors from a particular internet protocol (IP) address;
forwarding, by the edge processor, a reduced dataset relative to the data received from the data source to a data intake and query system, wherein the reduced dataset includes the first set of anomalies and the second set of anomalies;
generating a user interface that includes an indication of a likelihood of a presence of either the first set of anomalies or the second set of anomalies, wherein the user interface is configured for display on a computing device, and
blocking an identified internet protocol (IP) address associated with either the first set of anomalies or the second set of anomalies.
19 . The one or more non-transitory computer-readable media of claim 18 , further comprising instructions that, when executed by the one or more processors, cause the one or more processors to further perform the operations including, for each event in the set of events, adding a sampling field value, wherein:
sampling the subset of events comprises executing an object query specifying sampling criteria, where each event in the set of events has a sampling field value that meets the sampling criteria, and
the object query is associated with a data model representing a view of the set of events.
20 . The one or more non-transitory computer-readable media of claim 18 , wherein: an edge processor receives the data from the data source,
the edge processor executes at least a portion of the first set of anomaly detection operations, and
the edge processor executes at least a portion of the second set of anomaly detections operations.
21 . The computer-implemented method of claim 1 , wherein the first set of anomalies and the second set of anomalies each includes one or more anomalies.
22 . The one or more non-transitory computer-readable media of claim 18 , wherein the first set of anomalies and the second set of anomalies each includes one or more anomalies.