Granted Patent B2

US 12,292,874 · App. 18/536,592 · Granted May 6, 2025

Inventors: Miguel Casanova (Dublin, IE); David Tracey (Monasterboice, IE)

Assignee: Rapid7, Inc.

G06F16/2255G06F16/2264G06F16/24556G06F16/2477G06F16/285

Abstract

Claims (105)

1. A method comprising:

performing, by a computing system that implement having one or more hardware processors and associated memory:

receiving a plurality of events from a remote network and storing the events in an event log repository;

creating in memory a time-sliced approximate data structure (TSADS) for the events in the event log repository, wherein the TSADS includes a counts matrix and a statistics matrix and:

(a) the statistics matrix is used to store approximate statistics for different groups of timestamped datapoints in a plurality of time slices, and

(b) the counts matrix implements a count-min sketch to store approximate counts of datapoints in the different groups in the time slices;

receiving, via a query user interface, a query directed to the event log repository specifying to retrieve, from the TSADS, approximate statistics for a group of datapoints in the time slices, wherein the query specifies a group key of the group and a time range for retrieval;

responding to the query with approximate statistics for individual ones of the time slices, including:

selecting a set of cells in the count-min sketch in the counts matrix based on the group key and the time slice, wherein each cell in the set stores an approximate count of datapoints in the group in the time slice;

determining a first cell from the set that stores a best approximate count;

determining a best approximate statistic of the group in the time slice, wherein the best approximate statistic is retrieved from a second cell in the statistics matrix that corresponds to the first cell in the counts matrix; and

returning a time series of best approximate statistics of the group determined for each time slice, wherein the time series corresponds to the time range specified by the query.

2. The method of claim 1 , wherein:

the counts matrix is a three-dimensional matrix, wherein a first dimension represents a set of hash functions used to hash the group key, a second dimension represents respective hash spaces of the hash functions, and a third dimension represents the time slices; and

the statistics matrix has the same dimensions as the counts matrix.

3. The method of claim 1 , further comprising performing, by the computing system:

receiving a create request to create the TSADS, wherein the create request specifies a size of the counts matrix and the statistics matrix; and

allocating memory for the counts matrix and the statistics matrix according to the specified size.

4. The method of claim 1 , further comprising performing, by the computing system:

receiving an add request to add a datapoint to the TSADS, wherein the add request indicates a time of the datapoint, the group key of the datapoint, and a value of the datapoint;

in response to the add request:

selecting a first set of cells in the counts matrix based on the group key and the time;

updating the first set of cells to store new approximate counts;

determining a second set of cells in the statistics matrix that corresponds to the first set of cells in the counts matrix; and

updating the second set of cells to store new approximate statistics based on the value.

5. The method of claim 4 , wherein:

the datapoint is a compound datapoint that represents a plurality of datapoints, and the add request specifies a count of the datapoints represented; and

the updating of the first set of cells comprises incrementing previous values of the first set of cells by the specified count.

6. The method of claim 4 , wherein:

the computing system comprises a network of computing nodes including a first node and a second node;

the add request is performed by the first node;

the query correspond to a retrieve request performed by the second node; and

the method further comprises the first node transmitting the TSADS to the second node.

7. The method of claim 1 , wherein the computing system implements an event monitoring service and the method further comprises the event monitoring service:

monitoring the events stored in the event log repository for incidents associated with malware, phishing, or intrusion;

determining, based on the returned time series, a security incident in the remote network; and

generating an alert on a graphical user interface indicating the security incident.

8. The method of claim 7 , wherein the event monitoring service is implemented in a resource provider network that implements resource services for a plurality of clients, wherein the events are received from the remote network over a public network.

9. The method of claim 1 , wherein the counts matrix and the statistics matrix are implemented as two separate matrices in the memory of the computing system.

10. A system, comprising:

one or more hardware processors with associated memory that implement a computing system, configured to:

receive a plurality of events from a remote network and store the events in an event log repository;

create in memory a time-sliced approximate data structure (TSADS for the events in the event log repository, wherein the TSADS) includes a counts matrix and a statistics matrix and:

(a) the statistics matrix is used to store approximate statistics for different groups of timestamped datapoints in a plurality of time slices, and

(b) the counts matrix implements a count-min sketch to store approximate counts of datapoints in the different groups in the time slices;

receive, via a query user interface, a query directed to the event log repository specifying to retrieve, from the TSADS, approximate statistics for a group of datapoints in the time slices, wherein the query specifies a group key of the group and a time range for retrieval;

respond to the query with approximate statistics for individual ones of the time slices, including to:

select a set of cells in the count-min sketch in the counts matrix based on the group key and the time slice, wherein each cell in the set stores an approximate count of datapoints in the group in the time slice;

determine a first cell from the set that stores a best approximate count;

determine a best approximate statistic of the group in the time slice, wherein the best approximate statistic is retrieved from a second cell in the statistics matrix that corresponds to the first cell in the counts matrix; and

return a time series of best approximate statistics of the group determined for each time slice, wherein the time series corresponds to the time range specified by the query.

11. The system of claim 10 , wherein the computing system implements an event monitoring service configured to:

monitor the events stored in the event log repository for incidents associated with malware, phishing, or intrusion;

determine, based on the returned time series, a security incident in the remote network; and

generate an alert on a graphical user interface indicating the security incident.

12. The system of claim 10 , wherein:

the counts matrix as a three-dimensional matrix, wherein a first dimension represents the hash functions used to hash the group key, a second dimension represents respective hash spaces of the hash functions, and a third dimension represents the time slices; and

the statistics matrix has the same dimensions as the counts matrix.

13. The system of claim 10 , wherein the computing system is configured to:

select, at runtime, a summary function to for the TSADS to compute the approximate statistics in the statistics matrix, wherein the summary function is selected from a plurality of summary functions that compute different types of summary values for a group of data points, including two or more of:

a count of datapoints in the group,

a maximum of datapoint values in the group,

a minimum of datapoint values in the group,

an average of datapoint values in the group,

a sum of datapoint values in the group,

a maximum size of datapoints in the group,

a minimum size of datapoints in the group,

an average size of datapoints in the group, and

a total size of datapoints in the group.

14. The system of claim 10 , wherein the computing system is configured to:

receive a remove request to remove a particular group key from the TSADS, and in response:

select a first set of cells in the counts matrix based on the particular group key;

update the first set of cells to decrease approximate counts stored by the first set of cells by an approximate count of the particular group key;

determine a second set of cells in the statistics matrix that corresponds to the first set of cells; and

update approximate statistics stored by the second set of cells to remove approximate statistics of the particular group key.

15. The system of claim 10 , wherein the computing system is configured to:

in response to another query to retrieve collective counts and collective statistics for all groups in the TSADS:

determine, for each of the time slices, collective counts and collective statistics for all datapoints in the time slice as a single group;

return one or more time series of the collective counts and the collective statistics for each time slice.

16. The system of claim 10 , wherein the computing system is configured to:

in response to a merge request, perform a merge of the TSADS into another TSADS, wherein:

the two TSADSs store data for two distinct sets of time slices, and

the merge combines the two distinct sets of time slices in the other TSADS.

17. The system of claim 10 , wherein:

in response to a merge request, perform a merge of the TSADS into another TSADS, wherein:

the two TSADSs store data for two distinct sets of group keys, and

the merge combines the two distinct sets of group keys in the other TSADS.

18. One or more non-transitory computer-accessible storage media storing program instructions that when executed on or across one or more processors cause the one or more processors to:

receive a plurality of events from a remote network and store the events in an event log repository;

create in memory a time-sliced approximate data structure (TSADS) for the events in the event log repository, wherein the TSADS includes a counts matrix and a statistics matrix and:

(a) the statistics matrix is used to store approximate statistics for different groups of timestamped datapoints in a plurality of time slices, and

(b) the counts matrix implements a count-min sketch to store approximate counts of datapoints in the different groups in the time slices;

receive, via a query user interface, a query directed to the event log repository specifying to retrieve, from the TSADS, approximate statistics for a group of datapoints in the time slices, wherein the query specifies a group key of the group and a time range for retrieval;

respond to the query with approximate statistics for individual ones of the time slices, including to:

select a set of cells in the count-min sketch in the counts matrix based on the group key and the time slice, wherein each cell in the set stores an approximate count of datapoints in the group in the time slice;

determine a first cell from the set that stores a best approximate count;

determine a best approximate statistic of the group for the time slice, wherein the best approximate statistic is retrieved from a second cell in the statistics matrix that corresponds to the first cell in the counts matrix; and

return a time series of best approximate statistics of the group determined for each time slice, wherein the time series corresponds to the time range specified by the query.

19. The one or more non-transitory computer-accessible storage media of claim 18 , wherein the program instructions when executed on or across the one or more processors implement an event monitoring service and cause the event monitoring service to:

monitor the events stored in the event log repository for incidents associated with malware, phishing, or intrusion;

determine, based on the returned time series, a security incident in the remote network; and

generate an alert on a graphical user interface indicating the security incident.

20. The one or more non-transitory computer-accessible storage media of claim 18 , wherein the program instructions when executed on or across the one or more processors cause the one or more processors to:

implement the counts matrix as a three-dimensional matrix, wherein a first dimension represents a set of hash functions used to hash the group key, a second dimension represents respective hash spaces of the hash functions, and a third dimension represents the time slices; and

implement the statistics matrix to have the same dimensions as the counts matrix.

Assignments (2)

Continuity (2)

References Cited (25)