Advanced persistent threat detection by an information technology and security operations application
Techniques are described for enabling an IT and security operations application to detect and remediate advanced persistent threats (APTs). The detection of APTs involves the execution of search queries to search event data that initially was associated with lower-severity activity or that otherwise did not initially rise to the level of actionable event data in the application. The execution of such search queries may thus generally be configured to search non-real-time event data, e.g., event data that outside of a current window of days or a week and instead searches and aggregates event data spanning time periods of many weeks, months, or years. Due the nature of APTs, analyses of historical event data spanning such relatively long periods of time may in the aggregate uncover the types of persistent activity associated with APTs that would otherwise go undetected based only on searches of more current, real-time event data.
1. A computer-implemented method comprising:
executing a first query against timestamped event data stored in a data store to identify a first set of events, wherein the timestamped event data is generated by a data intake and query system based on raw machine data obtained from one or more components of an information technology (IT) environment, wherein the first query identifies events associated with a type of security incident, wherein events in the first set of events are assigned a first severity level by an IT and security operations application, wherein a second set of events stored in the data store do not satisfy the first query, and wherein events in the second set of events are assigned a second severity level that is less than the first severity level;
executing a second query against the second set of events that do not satisfy the first query to identify a third set of events, and wherein events in the third set of events do not individually indicate a security incident triggering an automated set of actions;
identifying, based on execution of the second query against the second set of events, a security event associated with an automated set of actions to be implemented by the IT and security operations application; and
implementing the automated set of actions.
2. The computer-implemented method of claim 1 , wherein the security event represents an advanced persistent threat (APT).
3. The computer-implemented method of claim 1 , wherein timestamps associated with events in the second set of events span a time period of at least a year.
4. The computer-implemented method of claim 1 , wherein the first query is a correlation search that the data intake and query system executes on a periodic basis and that is used to identify notable events in the timestamped event data.
5. The computer-implemented method of claim 1 , wherein implementing the automated set of actions includes generating a notification indicating an occurrence of the security event.
6. The computer-implemented method of claim 1 , wherein the timestamped event data is generated based on raw machine data obtained from an IT environment associated with a user of the IT and security operations application.
7. The computer-implemented method of claim 1 , wherein the IT and security operations application causes the data intake and query system to execute the second query, and wherein the IT and security operations application and the data intake and query system execute in a cloud provider network.
8. The computer-implemented method of claim 1 , wherein the first query is configured to identify event data indicating at least one of: a malware infection, a virus infection, a phishing attempt, a network-based attacked, or a failed login attempt.
9. The computer-implemented method of claim 1 , wherein the second query identifies the second set of events based on at least one event attribute that is shared by each event of the second set of events.
10. The computer-implemented method of claim 1 , wherein the second query identifies the second set of events based on at least one event attribute that is shared by each event of the second set of events, and wherein the method further includes: generating an aggregated value based on an aggregation of values contained in the events of the second set of events; and determining that the aggregated value exceeds a defined threshold.
11. The computer-implemented method of claim 1 , wherein the second query identifies the second set of events based on at least one event attribute that is shared by each event of the second set of events, and wherein identifying the security event includes determining that a type of activity indicated by events in the second set of events is recurring at a same computing device of an IT environment a threshold number of times over a defined period of time.
12. The computer-implemented method of claim 1 , wherein the second query identifies the second set of events based on at least one event attribute that is shared by each event of the second set of events, and wherein identifying the security event includes determining that a number of computing devices of an IT environment affected by a type of activity indicated by events in the second set of events is increasing at a threshold rate over a defined period of time.
13. The computer-implemented method of claim 1 , wherein the IT and security operations application causes the data intake and query system to execute the first query and the second query.
14. The computer-implemented method of claim 1 , wherein the IT and security operations application obtains data related to incidents from the data intake and query system running in a cloud provider network.
15. A system comprising:
a first electronic device to implement a data intake and query system, the data intake and query system including instructions that upon execution cause the data intake and query system to:
execute a first query against timestamped event data stored in a data store to identify a first set of events, wherein the timestamped event data is generated by the data intake and query system based on raw machine data obtained from one or more components of an information technology (IT) environment, wherein the first query identifies events associated with a type of security incident, wherein events in the first set of events are assigned a first severity level by an IT and security operations application, wherein a second set of events stored in the data store do not satisfy the first query, and wherein events in the second set of events are assigned a second severity level that is less than the first severity level, and
execute a second query against the second set of events that do not satisfy the first query to identify a third set of events, and wherein events in the third set of events do not individually indicate a security incident triggering an automated set of actions; and
a second one or more electronic devices to implement the IT and security operations application in a cloud provider network, the IT and security operations application including instructions that upon execution cause the IT and security operations application to:
identify, based on execution of the second query against the second set of events, a security event associated with an automated set of actions to be implemented by the IT and security operations application, and
implement the automated set of actions.
16. The system of claim 15 , wherein the security event represents an advanced persistent threat (APT).
17. The system of claim 15 , wherein timestamps associated with events in the second set of events span a time period of at least a year.
18. The system of claim 15 , wherein the first query is a correlation search that the data intake and query system executes on a periodic basis and that is used to identify notable events in the timestamped event data.