Selective event stream data storage based on network traffic volume
The disclosed embodiments provide a system that facilitates the processing of network data. During operation, the system causes for display a graphical user interface (GUI) for configuring the generation of time-series event data from network packets captured by one or more remote capture agents. Next, the system causes for display, in the GUI, a first set of user-interface elements containing a set of statistics associated with one or more event streams that comprise the time-series event data. The system then causes for display, in the GUI, one or more graphs comprising one or more values from the set of statistics. Finally, the system causes for display, in the GUI, a value of a statistic from the set of statistics based on a position of a cursor over the one or more graphs.
1. A computer-implemented method, comprising:
receiving an event stream from a remote capture agent, wherein the event stream includes timestamped event data generated by the remote capture agent based on network traffic monitored by the remote capture agent;
determining a volume of the network traffic from which the remote capture agent generated the timestamped event data;
determining a percentage of the event stream to store in one or more data stores based on the volume of the network traffic; and
causing a portion of the event stream to be stored in the one or more data stores according to the percentage.
2. The method of claim 1 , wherein the volume of the network traffic from which the remote capture agent generated the timestamped event data is a first volume of first network traffic as measured at a first point in time, wherein the percentage of the event stream is a first percentage, and wherein the method further comprises:
determining a second volume of second network traffic from which the remote capture agent generates timestamped event data as measured at a second point in time, wherein the second volume of second network traffic is greater than the first volume of first network traffic;
determining a second percentage of the event stream to store in the one or more data stores based on the second volume of the second network traffic, wherein the second percentage is less than the first percentage; and
causing a portion of the event stream to be stored in the one or more data stores according to the second percentage.
3. The method of claim 1 , wherein the volume of the network traffic from which the remote capture agent generated the timestamped event is a first volume of first network traffic as measured at a first point in time, wherein the percentage of the event stream is a first percentage, and wherein the method further comprises:
determining a second volume of second network traffic from which the remote capture agent generates timestamped event data as measured at a second point in time, wherein the second volume of second network traffic is less than the first volume of first network traffic;
determining a second percentage of the event stream to store in the one or more data stores based on the second volume of the second network traffic, wherein the second percentage is greater than the first percentage; and
causing a portion of the event stream to be stored in the one or more data stores according to the second percentage.
4. The method of claim 1 , further comprising generating statistical data based on the event stream without subsequently storing a portion of the event stream.
5. The method of claim 1 , wherein the event stream is generated by a plurality of remote capture agents distributed across a computer network.
6. The method of claim 1 , further comprising causing display of a graphical user interface (GUI) displaying a graph of statistical data generated based on the event stream.
7. The method of claim 1 , wherein the percentage of the event stream to store in the one more data stores is further based on a daily storage limit.
8. The method of claim 1 , wherein the percentage of the event stream to store in the one or more data stores is further based on an average amount of daily storage used by the event stream relative to a daily storage limit.
9. The method of claim 1 , wherein the percentage of the event stream to store in the one or more data stores is further based on an average amount of storage used by at least one other event stream generated by the remote capture agent.
10. The method of claim 1 , further comprising generating a statistic based on the event stream, wherein the statistic includes at least one of:
a total number of timestamped events in the event stream,
a total data size associated with incoming network traffic represented by the event stream,
a total data size associated with outgoing traffic represented by the event stream,
a total data size associated with network traffic represented by the event stream, and
an estimated index volume for each of the event stream.
11. The method of claim 1 , wherein the portion of the event stream is a first portion, and wherein causing the first portion of the event stream to be stored in the one or more data stores includes discarding a second portion of the event stream.
12. The method of claim 1 , wherein the percentage of the event stream to store in the one or more data stores is further based on a priority of the event stream relative to one or more other event streams.
13. The method of claim 1 , further comprising causing display of a graphical user interface (GUI) including at least one interface element used to manage the event stream, and wherein managing the event stream includes enabling selective storage of the event stream based on network traffic volume.
14. The method of claim 1 , further comprising periodically generating historical data indicating an amount of storage used by the event stream over one or more periods of time.
15. A computing device, comprising:
a processor; and
a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations including:
receiving an event stream from a remote capture agent, wherein the event stream includes timestamped event data generated by the remote capture agent based on network traffic monitored by the remote capture agent;
determining a volume of the network traffic from which the remote capture agent generated the timestamped event data;
determining a percentage of the event stream to store in one or more data stores based on the volume of the network traffic; and
causing a portion of the event stream to be stored in the one or more data stores according to the percentage.
16. The computing device of claim 15 , wherein the volume of the network traffic from which the remote capture agent generated the timestamped event data is a first volume of first network traffic as measured at a first point in time, wherein the percentage of the event stream is a first percentage, and wherein the instructions, when executed by the processor, further cause the processor to perform operations including:
determining a second volume of second network traffic from which the remote capture agent generates timestamped event data as measured at a second point in time, wherein the second volume of second network traffic is greater than the first volume of first network traffic;
determining a second percentage of the event stream to store in the one or more data stores based on the second volume of the second network traffic, wherein the second percentage is less than the first percentage; and
causing a portion of the event stream to be stored in the one or more data stores according to the second percentage.
17. The computing device of claim 15 , wherein the volume of the network traffic from which the remote capture agent generated the timestamped event is a first volume of first network traffic as measured at a first point in time, wherein the percentage of the event stream is a first percentage, and wherein the instructions, when executed by the processor, further cause the processor to perform operations including:
determining a second volume of second network traffic from which the remote capture agent generates timestamped event data as measured at a second point in time, wherein the second volume of second network traffic is less than the first volume of first network traffic;
determining a second percentage of the event stream to store in the one or more data stores based on the second volume of the second network traffic, wherein the second percentage is greater than the first percentage; and
causing a portion of the event stream to be stored in the one or more data stores according to the second percentage.
18. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform operations including:
receiving an event stream from a remote capture agent, wherein the event stream includes timestamped event data generated by the remote capture agent based on network traffic monitored by the remote capture agent;
determining a volume of the network traffic from which the remote capture agent generated the timestamped event data;
determining a percentage of the event stream to store in one or more data stores based on the volume of the network traffic; and
causing a portion of the event stream to be stored in the one or more data stores according to the percentage.
19. The non-transitory computer-readable medium of claim 18 , wherein the volume of the network traffic from which the remote capture agent generated the timestamped event data is a first volume of first network traffic as measured at a first point in time, wherein the percentage of the event stream is a first percentage, and wherein the instructions, when executed by the one or more processors, further cause the one or more processors to perform operations including:
determining a second volume of second network traffic from which the remote capture agent generates timestamped event data as measured at a second point in time, wherein the second volume of second network traffic is greater than the first volume of first network traffic;
determining a second percentage of the event stream to store in the one or more data stores based on the second volume of the second network traffic, wherein the second percentage is less than the first percentage; and
causing a portion of the event stream to be stored in the one or more data stores according to the second percentage.
20. The non-transitory computer-readable medium of claim 18 , wherein the volume of the network traffic from which the remote capture agent generated the timestamped event is a first volume of first network traffic as measured at a first point in time, wherein the percentage of the event stream is a first percentage, and wherein the instructions, when executed by the one or more processors, further cause the one or more processors to perform operations including:
determining a second volume of second network traffic from which the remote capture agent generates timestamped event data as measured at a second point in time, wherein the second volume of second network traffic is less than the first volume of first network traffic;
determining a second percentage of the event stream to store in the one or more data stores based on the second volume of the second network traffic, wherein the second percentage is greater than the first percentage; and
causing a portion of the event stream to be stored in the one or more data stores according to the second percentage.