Application-based configuration of network data capture by remote capture agents
The disclosed embodiments provide a method and system for facilitating the processing of network data. During operation, the system obtains, at a remote capture agent, configuration information for the remote capture agent from a configuration server over a network. Next, the system uses the configuration information to configure the generation of event data from network packets at the remote capture agent. Upon receiving an update to the configuration information from the configuration server, the system uses the update to reconfigure the generation of the event data by the remote capture agent during runtime of the remote capture agent.
1. A computer-implemented method performed by a configuration server coupled to a network, the method comprising:
obtaining configuration information generated and sent to the configuration server by an application running on a server that is separate from the configuration server and that is coupled to the configuration server via the network, the configuration information specifying one or more event streams to be generated by one or more remote capture agents, the one or more event streams including timestamped event data generated by the one or more remote capture agents based on network traffic monitored by the one or more remote capture agents, and wherein the one or more remote capture agents are installed in a virtual computing environment; and
sending the configuration information to the one or more remote capture agents, the configuration information causing the one or more remote capture agents to:
generate the one or more event streams including timestamped event data generated based on network traffic monitored by the one or more remote capture agents, and
send the one or more event streams to another component on the network for storage in a data store accessible to the application.
2. The computer-implemented method of claim 1 , wherein the configuration information causes the one or more remote capture agents to generate each event of the timestamped event data by:
identifying boundaries of the event in the network traffic;
extracting network packet data from at least one network packet of the network traffic and associating the network packet data with the event;
determining a timestamp associated with the event; and
associating the timestamp with the event.
3. The computer-implemented method of claim 1 , wherein the configuration information further causes the one or more remote capture agents to generate each event of the timestamped event data by:
extracting network packet data from at least one network packet of the network traffic and associating the network packet data with the event;
applying a filtering rule to the network packet data to determine an event type associated with the event; and
adding, based on the determined event type, the event to at least one event stream of one or more event streams.
4. The computer-implemented method of claim 1 , wherein the application uses the one or more event streams to satisfy search requests for data contained in the one or more event streams.
5. The computer-implemented method of claim 1 , wherein the application uses the one or more event streams to generate visualizations based on timestamped event data contained in the one or more event streams.
6. The computer-implemented method of claim 1 , wherein the configuration information specifies one or more transformations to be applied to the timestamped event data, and wherein the one or more transformations comprise at least one of an aggregation, a calculation, a filter, a normalization, and a formatting.
7. The computer-implemented method of claim 1 , wherein the configuration information comprises at least one of an identifier for an event stream, a description for the event stream, an event stream type for the event stream, a custom field for the event stream, and an additional parameter for the event stream.
8. The computer-implemented method of claim 1 , wherein the configuration information comprises an additional parameter, and wherein the additional parameter is at least one of a time interval between events, a maximum number of aggregated events, and an inclusion of a matching transaction or matching error in the event data.
9. A configuration server coupled to a network, the configuration server comprising:
a processor;
a non-transitory computer readable storage medium storing instructions which, when executed by the processor, cause the configuration server to:
obtain configuration information generated and sent to the configuration server by an application running on a server that is separate from the configuration server and that is coupled to the configuration server via the network, the configuration information specifying one or more event streams to be generated by one or more remote capture agents, the one or more event streams including timestamped event data generated by the one or more remote capture agents based on network traffic monitored by the one or more remote capture agents, and wherein the one or more remote capture agents are installed in a virtual computing environment; and
send the configuration information to the one or more remote capture agents, the configuration information causing the one or more remote capture agents to:
generate the one or more event streams including timestamped event data generated based on network traffic monitored by the one or more remote capture agents, and
send the one or more event streams to another component on the network for storage in a data store accessible to the application.
10. The configuration server of claim 9 , wherein the configuration information further causes the one or more remote capture agents to generate each even of the timestamped event data by:
identifying boundaries of the event in the network traffic;
extracting network packet data from at least one network packet of the network traffic and associating the network packet data with the event;
determining a timestamp associated with the event; and
associating the timestamp with the event.
11. The configuration server of claim 9 , wherein the configuration information further causes the one or more remote capture agents to generate each event of the timestamped event data by:
extracting network packet data from at least one network packet of the network traffic and associating the network packet data with the event;
applying a filtering rule to the network packet data to determine an event type associated with the event; and
adding, based on the determined event type, the event to at least one event stream of one or more event streams.
12. The configuration server of claim 9 , wherein the application uses the one or more event streams to satisfy search requests for data contained in the one or more event streams.
13. The configuration server of claim 9 , wherein the application uses the one or more event streams to generate visualizations based on timestamped event data contained in the one or more event streams.
14. The configuration server of claim 9 , wherein the configuration information specifies one or more transformations to be applied to the timestamped event data, and wherein the one or more transformations comprise at least one of an aggregation, a calculation, a filter, a normalization, and a formatting.
15. The configuration server of claim 9 , wherein the configuration information comprises at least one of an identifier for an event stream, a description for the event stream, an event stream type for the event stream, a custom field for the event stream, and an additional parameter for the event stream.
16. The configuration server of claim 9 , wherein the configuration information comprises an additional parameter, and wherein the additional parameter is at least one of a time interval between events, a maximum number of aggregated events, and an inclusion of a matching transaction or matching error in the event data.
17. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause a configuration server to perform operations comprising:
obtaining configuration information generated and sent to the configuration server by an application running on a server that is separate from the configuration server and that is coupled to the configuration server via the network, the configuration information specifying one or more event streams to be generated by one or more remote capture agents, the one or more event streams including timestamped event data generated by the one or more remote capture agents based on network traffic monitored by the one or more remote capture agents, and wherein the one or more remote capture agents are installed in a virtual computing environment; and
sending the configuration information to the one or more remote capture agents, the configuration information causing the one or more remote capture agents to:
generate the one or more event streams including timestamped event data generated based on network traffic monitored by the one or more remote capture agents, and
send the one or more event streams to another component on the network for storage in a data store accessible to the application.
18. The non-transitory computer-readable storage medium of claim 17 , wherein the configuration information causes the one or more remote capture agents to generate each event of the timestamped event data by:
identifying boundaries of the event in the network traffic;
extracting network packet data from at least one network packet of the network traffic and associating the network packet data with the event;
determining a timestamp associated with the event; and
associating the timestamp with the event.
19. The non-transitory computer-readable storage medium of claim 17 , wherein the configuration information further causes the one or more remote capture agents to generate each event of the timestamped event data by:
extracting network packet data from at least one network packet of the network traffic and associating the network packet data with the event;
applying a filtering rule to the network packet data to determine an event type associated with the event; and
adding, based on the determined event type, the event to at least one event stream of one or more event streams.
20. The non-transitory computer-readable storage medium of claim 17 , wherein the application uses the one or more event streams to satisfy search requests for data contained in the one or more event streams.
21. The non-transitory computer-readable storage medium of claim 17 , wherein the application uses the one or more event streams to generate visualizations based on timestamped event data contained in the one or more event streams.
22. The non-transitory computer-readable storage medium of claim 17 , wherein the configuration information specifies one or more transformations to be applied to the timestamped event data, and wherein the one or more transformations comprise at least one of an aggregation, a calculation, a filter, a normalization, and a formatting.
23. The non-transitory computer-readable storage medium of claim 17 , wherein the configuration information comprises at least one of an identifier for an event stream, a description for the event stream, an event stream type for the event stream, a custom field for the event stream, and an additional parameter for the event stream.
24. The non-transitory computer-readable storage medium of claim 17 , wherein the configuration information comprises an additional parameter, and wherein the additional parameter is at least one of a time interval between events, a maximum number of aggregated events, and an inclusion of a matching transaction or matching error in the event data.