Configuring generation of event streams by remote capture agents
The disclosed embodiments provide a system that processes network data. During operation, the system obtains, at a remote capture agent, a first protocol classification for a first packet flow captured by the remote capture agent. Next, the system uses configuration information associated with the first protocol classification to build a first event stream from the first packet flow at the remote capture agent, wherein the first event stream comprises time-series event data generated from network packets in the first packet flow based on the first protocol classification. The system then transmits the first event stream over a network for subsequent storage and processing of the first event stream by one or more components on the network.
1. A method performed by a configuration server to generate a plurality of event streams from network packets monitored by a plurality of remote capture agents, the method comprising:
receiving, by the configuration server, input indicating:
first configuration data associated with a first event stream to be generated by a first remote capture agent of the plurality of remote capture agents, the first event stream associated with a first type of event and to include time-series event data representing instances of the first type of event in network packets monitored by the first remote capture agent, and
second configuration data associated with a second event stream to be generated by a second remote capture agent of the plurality of remote capture agents, the second event stream associated with a second type of event and to include time-series event data representing instances of the second type of event in the network packets monitored by the second remote capture agent; and
sending, over a network, the first configuration data to the first remote capture agent and the second configuration data to the second remote capture agent.
2. The method of claim 1 , wherein sending the first configuration data causes the first remote capture agent to configure generation of the time-series event data from the network packets during runtime of the first remote capture agent.
3. The method of claim 1 , further comprising receiving, by the configuration server, input indicating third configuration data associated with a third event stream to be generated by the first remote capture agent of the plurality of remote capture agents, the third event stream associated with a third type of event that is different from the first and second type of event and to include time-series event data representing instances of the third type of event in the network packets monitored by the first remote capture agent.
4. The method of claim 1 , wherein the first configuration data instructs the first remote capture agent to send the first event stream to another component on the network for subsequent processing.
5. The method of claim 1 , wherein the first configuration data instructs the first remote capture agent to, in response to detecting encryption of the network packets of the first event stream, decrypt the network packets prior to generating the first event stream.
6. The method of claim 1 , wherein each network packet of the network packets monitored by the first remote capture agent is associated with at least one of:
a source;
a destination;
a network address;
a port; and
a transport layer protocol.
7. The method of claim 1 , wherein the first configuration data further identifies one or more event attributes associated with the first type of event, the identified one or more event attributes causing the first remote capture agent to extract one or more values associated with the one or more event attributes from the network packets and to include the extracted one or more values in the first event stream.
8. The method of claim 1 , wherein the first configuration data further identifies one or more event attributes associated with the first type of event, the identified one or more event attributes causing the first remote capture agent to extract one or more values associated with the one or more event attributes from the network packets and to include the extracted one or more values in the first event stream, and wherein the first configuration data further instructs the first remote capture agent to perform one or more transformations to the extracted one or more values included in the first event stream.
9. The method of claim 1 , wherein the first type of event is associated with at least one of:
a transport layer protocol;
a session layer protocol;
a presentation layer protocol; and
an application layer protocol.
10. The method of claim 1 , wherein at least one of the plurality of remote capture agents is installed in a cloud computing environment.
11. An apparatus, comprising:
a processor;
a non-transitory computer readable storage medium storing instructions which, when
executed by the processor, cause the apparatus to:
receive, by a configuration server, input indicating:
first configuration data associated with a first event stream to be generated by a first remote capture agent of a plurality of remote capture agents, the first event stream associated with a first type of event and to include time-series event data representing instances of the first type of event in network packets monitored by the first remote capture agent, and
second configuration data associated with a second event stream to be generated by a second remote capture agent of the plurality of remote capture agents, the second event stream associated with a second type of event and to include time-series event data representing instances of the second type of event in the network packets monitored by the second remote capture agent; and
send, over a network, the first configuration data to the first remote capture agent and the second configuration data to the second remote capture agent.
12. The apparatus of claim 11 , wherein sending the first configuration data causes the first remote capture agent to configure generation of the time-series event data from the network packets during runtime of the first remote capture agent.
13. The apparatus of claim 11 , wherein the instructions, when executed by the processor, further cause the apparatus to receive, by the configuration server, input indicating third configuration data associated with a third event stream to be generated by the first remote capture agent of the plurality of remote capture agents, the third event stream associated with a third type of event that is different from the first and second type of event and to include time-series event data representing instances of the third type of event in the network packets monitored by the first remote capture agent.
14. The apparatus of claim 11 , wherein the first configuration data instructs the first remote capture agent to send the first event stream to another component on the network for subsequent processing.
15. The apparatus of claim 11 , wherein the first configuration data instructs the first remote capture agent to, in response to detecting encryption of the network packets of the first event stream, decrypt the network packets prior to generating the first event stream.
16. The apparatus of claim 11 , wherein each network packet of the network packets monitored by the first remote capture agent is associated with at least one of:
a source;
a destination;
a network address;
a port; and
a transport layer protocol.
17. The apparatus of claim 11 , wherein the first configuration data further identifies one or more event attributes associated with the first type of event, the identified one or more event attributes causing the first remote capture agent to extract one or more values associated with the one or more event attributes from the network packets and to include the extracted one or more values in the first event stream.
18. The apparatus of claim 11 , wherein the first configuration data further identifies one or more event attributes associated with the first type of event, the identified one or more event attributes causing the first remote capture agent to extract one or more values associated with the one or more event attributes from the network packets and to include the extracted one or more values in the first event stream, and wherein the first configuration data further instructs the first remote capture agent to perform one or more transformations to the extracted one or more values included in the first event stream.
19. The apparatus of claim 11 , wherein the first type of event is associated with at least one of:
a transport layer protocol;
a session layer protocol;
a presentation layer protocol; and
an application layer protocol.
20. The apparatus of claim 11 , wherein at least one of the plurality of remote capture agents is installed in a cloud computing environment.
21. A non-transitory computer-readable storage medium storing instructions which, when executed by a processor, cause the processor to perform operations comprising:
receiving, by a configuration server, input indicating:
first configuration data associated with a first event stream to be generated by a first remote capture agent of a plurality of remote capture agents, the first event stream associated with a first type of event and to include time-series event data representing instances of the first type of event in network packets monitored by the first remote capture agent, and
second configuration data associated with a second event stream to be generated by a second remote capture agent of the plurality of remote capture agents, the second event stream associated with a second type of event and to include time-series event data representing instances of the second type of event in the network packets monitored by the second remote capture agent; and
sending, over a network, the first configuration data to the first remote capture agent and the second configuration data to the second remote capture agent.
22. The non-transitory computer-readable storage medium of claim 21 , wherein sending the first configuration data causes the first remote capture agent to configure generation of the time-series event data from the network packets during runtime of the first remote capture agent.
23. The non-transitory computer-readable storage medium of claim 21 , wherein the instructions, when executed by the processor, further cause operations comprising receiving, by the configuration server, input indicating third configuration data associated with a third event stream to be generated by the first remote capture agent of the plurality of remote capture agents, the third event stream associated with a third type of event that is different from the first and second type of event.
24. The non-transitory computer-readable storage medium of claim 21 , wherein the first configuration data instructs the first remote capture agent to send the first event stream to another component on the network for subsequent processing.
25. The non-transitory computer-readable storage medium of claim 21 , wherein the first configuration data instructs the first remote capture agent to, in response to detecting encryption of the network packets of the first event stream, decrypt the network packets prior to generating the first event stream.
26. The non-transitory computer-readable storage medium of claim 21 , wherein each network packet of the network packets monitored by the first remote capture agent is associated with at least one of:
a source;
a destination;
a network address;
a port; and
a transport layer protocol.
27. The non-transitory computer-readable storage medium of claim 21 , wherein the first configuration data further identifies one or more event attributes associated with the first type of event, the identified one or more event attributes causing the first remote capture agent to extract one or more values associated with the one or more event attributes from the network packets and to include the extracted one or more values in the first event stream.
28. The non-transitory computer-readable storage medium of claim 21 , wherein the first configuration data further identifies one or more event attributes associated with the first type of event, the identified one or more event attributes causing the first remote capture agent to extract one or more values associated with the one or more event attributes from the network packets and to include the extracted one or more values in the first event stream, and wherein the first configuration data further instructs the first remote capture agent to perform one or more transformations to the extracted one or more values included in the first event stream.
29. The non-transitory computer-readable storage medium of claim 21 , wherein the first type of event is associated with at least one of:
a transport layer protocol;
a session layer protocol;
a presentation layer protocol; and
an application layer protocol.
30. The non-transitory computer-readable storage medium of claim 21 , wherein at least one of the plurality of remote capture agents is installed in a cloud computing environment.