Throttling system and method
The flow of events though an event-analysis system is controlled by a number of event throttles which filter events, prioritize events and control the rate at which events are provided to event-processing components of the event-analysis system. Incoming events to the event-analysis system are associated with a profile, and a metrics engine generates metrics based on the incoming events for each profile. The flow of events to the metrics engine is controlled on a per profile basis, so that excessive generation of new metrics and new profiles is limited. If the system from which the events originate is compromised, metrics associated with compromised profiles may be frozen to avoid corrupting existing metrics. Processing of events and anomalies by analysis engines within the event-analysis system may be delayed to allow the accumulation of metrics necessary for accurate analysis.
1. A computer-implemented method, comprising:
generating, based at least in part on an event obtained from an event stream, a metric associated with a profile;
determining that a number of metrics generated over a period of time does not exceed a threshold number of metrics for the profile; and
as a result of determining that the number of metrics does not exceed the threshold number of metrics for the profile, adding the metric to a set of metrics associated with the profile.
2. The computer-implemented method of claim 1 , further comprising:
generating, based at least in part on an additional event obtained from the event stream, an additional metric associated with the profile;
determining that the number of metrics generated over the period of time exceeds the threshold number of metrics for the profile; and
as a result of determining that the number of metrics exceeds the threshold number of metrics for the profile, discarding the additional metric.
3. The computer-implemented method of claim 2 , further comprising:
selecting the event from the event stream based on a priority associated with the event; and
selecting the additional event from the event stream based at least in part on the priority of the additional event.
4. The computer-implemented method of claim 1 , further comprising:
identifying a plurality of events in the event stream, individual events of the plurality of events including a matching event type;
generating a combined event based on the plurality of events, the combined event including an occurrence count for the combined event; and
replacing the plurality of events with the combined event in the event stream.
5. A computer system, comprising:
one or more processors; and
a memory storing executable instructions that, as a result of being executed by the one or more processors, cause the computer system to:
generate, based at least in part on an event obtained from an event stream, a metric for an event type associated with the event;
determine that a number of metrics for the event type that are generated over a period of time does not exceed a threshold number of metrics; and
as a result of determining that the number of metrics does not exceed the threshold number of metrics, store the metric.
6. The computer system of claim 5 , wherein the executable instructions, as a result of being executed, further cause the computer system to:
determine that the number of metrics for the event type that are generated over a period of time exceeds a threshold number of metrics; and
as a result of determining that the number of metrics exceeds the threshold number of metrics, discard the metric.
7. The computer system of claim 5 , wherein the threshold number of metrics is determined based on an amount of storage space allocated to storing metrics.
8. The computer system of claim 5 , wherein the executable instructions, as a result of being executed, further cause the computer system to determine a level of confidence for the metric based on a standard deviation of a metric associated with the event type.
9. The computer system of claim 5 , wherein the executable instructions, as a result of being executed, further cause the computer system to:
determine that a threshold number of events have been obtained; and
as a result of determining that the threshold number of events have been obtained, determine whether the event indicates an anomaly.
10. The computer system of claim 9 , wherein the executable instructions, as a result of being executed, further cause the computer system to:
determine that the event indicates a critical anomaly; and
as a result of determining that a critical anomaly is indicated, prevent changes to one or more metrics associated with the event type.
11. The computer system of claim 9 , wherein the executable instructions, as a result of being executed, further cause the computer system to, as a result of determining that a critical anomaly is indicated, deletes one or more metrics associated with the event type.
12. The computer system of claim 5 , wherein the executable instructions, as a result of being executed, further cause the computer system to:
confirm that the event was generated by a trusted source; and
as a result of confirming that the event was generated by a trusted source, delete the event.
13. A non-transitory computer-readable storage medium with executable instructions stored thereon that, as a result of being executed by one or more processors of a computer system, cause the computer system to at least:
obtain, over a period of time, a set of event records from an event source;
identify a set of profiles associated with individual event records in the set of event records; and
generate a set of metrics for individual profiles in the set of profiles, such that a number of metrics for the individual profiles in the set of profiles is limited to a threshold number of metrics over the period of time.
14. The non-transitory computer-readable storage medium of claim 13 , wherein the executable instructions that cause the computer system to obtain the set of event records from an event source further include executable instructions that cause the computer system to:
obtain a first event and a second event from the event source;
determine that the first event and the second event are of a matching type; and
combine the first event and the second event to produce the event record.
15. The non-transitory computer-readable storage medium of claim 13 , wherein:
the event source is an event queue maintained by the computer system; and
a minimum number of events are maintained in the event queue.
16. The non-transitory computer-readable storage medium of claim 13 , wherein the executable instructions further comprise instructions that, as a result of being executed by the one or more processors, cause the computer system to:
maintain a burst bucket in association with a particular profile, the burst bucket including an associated bucket level adjusted in response to obtaining metrics with the particular profile; and
a limitation associated with the particular profile based at least in part on the bucket level of the burst bucket.
17. The non-transitory computer-readable storage medium of claim 13 , wherein the executable instructions further comprise instructions that, as a result of being executed by the one or more processors, cause the computer system to identify a set of anomalies based at least in part on the set of metrics.
18. The non-transitory computer-readable storage medium of claim 13 , wherein the executable instructions further comprise instructions that, as a result of being executed by the one or more processors, cause the computer system to display an interface that allows a user to:
select a particular profile from a number of profiles, wherein the particular profile is associated with a particular type of anomaly; and
select a parent-profile control element to change the selected profile to a parent profile of the particular profile within a profile hierarchy.
19. The non-transitory computer-readable storage medium of claim 18 , wherein the executable instructions further comprise instructions that, as a result of being executed by the one or more processors, cause the computer system to, as a result of the user selecting the particular profile, presenting a history of anomalies for the particular profile.
20. The non-transitory computer-readable storage medium of claim 13 , wherein the threshold number of metrics for a profile is based on a total number of metrics generated.