State-based detection of anomalous API calls within a real-time data stream
Media, methods, and systems are disclosed for detecting anomalous API calls within a real-time data flow. A real-time data stream is received. The real-time data stream is filtered. A stored state is modified based on a log entry data element within the filtered real-time data stream. An API call associated with the modified stored state is detected to be anomalous. Finally, a remediation action is taken for the API call.
1. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by a processor, perform a method for detecting anomalous data within a real-time data stream, the method comprising:
receiving the real-time data stream from a real-time data source, the real-time data stream comprising a plurality of log entries, individual log entries of the plurality of log entries including a plurality of data elements, wherein at least one data element of the plurality of data elements is obtained from an application programming interface (API) call, the real-time data source being a group-based communication system and the real-time data stream including group-based communication data associated with the group-based communication system, the group-based communication data including activity within communication channels by users that are authorized to access the communication channels;
filtering, as the real-time data stream is being received, the real-time data stream based at least in part on one or more filter criteria to obtain a filtered real-time data stream comprising a subset of the plurality of log entries;
generating a modified stored state by modifying a stored state corresponding to one or more log entry data elements of a particular type based at least in part on a determination that a log entry of the subset of the plurality of log entries is of the particular type, wherein the modified stored state corresponds to one or more failed log-in attempts by a particular user of the users of the group-based communication system and the particular type corresponds to a count of the one or more failed log-in attempts;
determining whether the count meets or exceeds a predetermined threshold;
determining that the modified stored state is an anomalous state based at least in part on the count meeting or exceeding the predetermined threshold; and
in response to determining that the modified stored state is the anomalous state, taking one or more remediation actions.
2. The one or more non-transitory computer-readable media of claim 1 , wherein the method further comprises:
receiving enriching data from an enriching data source, wherein the enriching data source is an external data feed;
prior to modifying the stored state and based at least in part on one or more enriching rules, modifying the one or more log entry data elements using the enriching data to obtain one or more enriched log entry data elements; and
updating the stored state based at least in part on the one or more enriched log entry data elements.
3. The one or more non-transitory computer-readable media of claim 1 , wherein the one or more remediation actions comprise sending information regarding the anomalous state to an external system.
4. The one or more non-transitory computer-readable media of claim 1 , wherein the one or more remediation actions comprise sending a message via the group-based communication system.
5. The one or more non-transitory computer-readable media of claim 1 , wherein the anomalous state signifies a first potential security threat, and wherein the API call associated with each of the plurality of data elements are also determined to be the first potential security threat or a second potential security threat.
6. The one or more non-transitory computer-readable media of claim 1 , wherein the stored state is associated with a particular user.
7. The one or more non-transitory computer-readable media of claim 1 , wherein the filtering of the real-time data stream comprises:
clustering, using a machine learning model, the plurality of log entries into a plurality of clusters that each include different groups of log entries of the plurality of log entries; and
generating the filtered real-time data stream by:
filtering out at least a first set of clusters of the plurality of clusters that are determined to have a first relevance, wherein the first set of clusters are not retained in memory; and
retaining, in the memory, a second set of clusters of the plurality of clusters that are determined to have a second relevance that is greater than the first relevance.
8. The one or more non-transitory computer-readable media of claim 1 , wherein the method further comprises:
partitioning the plurality of log entries into a first set of log entries that are associated with a first set of user identifiers (IDs) of a first set of users, and into a second set of log entries that are associated with a second set of user IDs of a second set of users, the first set of users being of a first type and the second set of users being of a second type;
assigning the first set of log entries to a first processor that is to process the first set of log entries; and
assigning the second set of log entries to a second processor, different than the first processor, that is to process the second set of log entries, the first processor to process first subsequent log entries associated with the first type and the second processor to process second subsequent log entries associated with the second type.
9. The one or more non-transitory computer-readable media of claim 1 , wherein the method further comprises generating the modified stored state by modifying the stored state corresponding to the one or more log entry data elements of the particular type based at least in part on enriching data that includes geolocation data associated with the log entry, the geolocation data being based at least in part on a mapping between an IP address associated with the log entry and a physical location associated with the log entry.
10. A method for monitoring anomalous data within a real-time data stream, the method comprising:
receiving the real-time data stream from a real-time data source, the real-time data stream comprising a plurality of log entries, individual log entries of the plurality of log entries including a plurality of data elements, wherein each at least one data element of the plurality of data elements is obtained from an application programming interface (API) call, the real-time data source being a group-based communication system and the real-time data stream including group-based communication data associated with the group-based communication system, the group-based communication data including activity within communication channels by users that are authorized to access the communication channels;
filtering, as the real-time data stream is being received, the real-time data stream based at least in part on one or more filter criteria to obtain a filtered real-time data stream comprising a subset of the plurality of log entries;
generating a modified stored state by modifying a stored state corresponding to one or more log entry data elements of a particular type based at least in part on a determination that a log entry of the subset of the plurality of log entries is of the particular type, wherein the modified stored state corresponds to one or more failed log-in attempts by a particular user of the users of the group-based communication system and the particular type corresponds to a count of the one or more failed log-in attempts;
determining whether the count meets or exceeds a predetermined threshold;
determining that the modified stored state is an anomalous state based at least in part on the count meeting or exceeding the predetermined threshold; and
in response to determining that the modified stored state is the anomalous state, taking one or more remediation actions.
11. The method of claim 10 , wherein the method further comprises:
receiving enriching data from an enriching data source, wherein the enriching data source is an external data feed;
prior to modifying the stored state and based at least in part on one or more enriching rules, modifying the one or more log entry data elements using the enriching data to obtain one or more enriched log entry data elements; and
updating the stored state based at least in part on the one or more enriched log entry data elements.
12. The method of claim 10 , wherein the one or more remediation actions comprise sending information regarding the anomalous state to an external system.
13. The method of claim 10 , wherein the one or more remediation actions comprise sending a message via the group-based communication system.
14. The method of claim 10 , wherein the anomalous state signifies a first potential security threat, and wherein the API call associated with each of the plurality of data elements are also determined to be the first potential security threat or a second potential security threat.
15. The method of claim 10 , wherein the stored state is associated with a particular user.
16. A system comprising at least one processor and at least one non-transitory memory storing computer-executable instructions that when executed by the at least one processor cause the system to carry out actions comprising:
receiving a real-time data stream from a real-time data source, the real-time data stream comprising a plurality of log entries, individual log entries of the plurality of log entries including a plurality of data elements, wherein each at least one data element of the plurality of data elements is obtained from an application programming interface (API) call, the real-time data source being a group-based communication system and the real-time data stream including group-based communication data associated with the group-based communication system, the group-based communication data including activity within communication channels by users that are authorized to access the communication channels;
filtering, as the real-time data stream is being received, the real-time data stream based at least in part on one or more filter criteria to obtain a filtered real-time data stream comprising a subset of the plurality of log entries;
generating a modified stored state by modifying a stored state corresponding to one or more log entry data elements of a particular type based at least in part on a determination that a log entry of the subset of the plurality of log entries is of the particular type, wherein the modified stored state corresponds to one or more failed log-in attempts by a particular user of the users of the group-based communication system and the particular type corresponds to a count of the one or more failed log-in attempts;
determining whether the count meets or exceeds a predetermined threshold;
determining that the modified stored state is an anomalous state based at least in part on the count meeting or exceeding the predetermined threshold; and
in response to determining that the modified stored state is the anomalous state, taking one or more remediation actions.
17. The system of claim 16 , wherein the actions further comprise:
receiving enriching data from an enriching data source, wherein the enriching data source is an external data feed;
prior to modifying the stored state and based at least in part on one or more enriching rules, modifying the one or more log entry data elements using the enriching data to obtain one or more enriched log entry data elements; and
updating the stored state based at least in part on the one or more enriched log entry data elements.
18. The system of claim 16 , wherein the one or more remediation actions comprise sending information regarding the anomalous state to an external system.
19. The system of claim 16 , wherein the one or more remediation actions comprise sending a message via the group-based communication system.
20. The system of claim 16 , wherein the anomalous state signifies a first potential security threat, and wherein the API call associated with each of the plurality of data elements are also determined to be the first potential security threat or a second potential security threat.