Transforming event data generated by remote capture agents using user-generated code
The disclosed embodiments provide a method and system for processing network data. During operation, the system obtains, at a remote capture agent, configuration information for the remote capture agent from a configuration server over a network. Next, the system uses the configuration information to configure the generation of event data from network data obtained from network packets at the remote capture agent. The system then uses the configuration information to configure transformation of the event data or the network data into transformed event data at the remote capture agent.
1. A computer-implemented method, comprising:
receiving, by a configuration server, input configuring a transformation to be applied to event data of an event stream generated by a remote capture agent, wherein the input configuring the transformation includes user-provided executable source code that, upon execution, performs an operation to transform the event data into transformed event data;
receiving, from the remote capture agent, the event stream including the event data, wherein the remote capture agent generates the event data based on network data monitored by the remote capture agent;
executing the user-provided executable source code to transform the event data into transformed event data; and
sending the transformed event data to another component for subsequent processing.
2. The computer-implemented method of claim 1 , further comprising sending the transformed event data to a transformation server that performs at least one additional transformation to the transformed event data.
3. The computer-implemented method of claim 1 , further comprising:
generating, by the configuration server, configuration information based at least in part on the input configuring the transformation; and
sending, to the remote capture agent, the configuration information, wherein the remote capture agent generates the event stream based on configuration information obtained by the remote capture agent from a configuration server.
4. The computer-implemented method of claim 1 , wherein the remote capture agent generates the event stream based on configuration information obtained by the remote capture agent from the configuration server, and wherein the method further comprises:
receiving an update to the configuration information to obtain updated configuration information; and
sending the updated configuration information to the remote capture agent, wherein the remote capture agent uses the updated configuration information to generate additional event data.
5. The computer-implemented method of claim 1 , further comprising: further transforming the event data into the transformed event data by performing at least one of: an aggregation, a calculation, a filter, a normalization, or a formatting.
6. The computer-implemented method of claim 1 , further comprising:
further transforming the event data by:
obtaining a time interval associated with the event data; and
aggregating the event data within the time interval into at least one of: an event count, a statistic, or a uniqueness count.
7. The computer-implemented method of claim 1 , wherein the remote capture agent generates the event data based on configuration information obtained by the remote capture agent from the configuration server, and wherein the configuration server obtains the configuration information from an application used to access the transformed event data.
8. The computer-implemented method of claim 1 , wherein the user-provided executable source code is specified using an interpreted programming language.
9. The computer-implemented method of claim 1 , wherein the remote capture agent is installed in a virtual computing environment.
10. The computer-implemented method of claim 1 , wherein the user-provided executable source code causes the transformed event data to be stored at a storage location specified by the user-provided executable source code.
11. A configuration server, comprising:
a processor; and
a memory storing instructions which, when executed by the processor, cause the configuration server to:
receive input configuring a transformation to be applied to event data of an event stream generated by a remote capture agent, wherein the input configuring the transformation includes user-provided executable source code that, upon execution, performs an operation to transform the event data into transformed event data;
receiving, from the remote capture agent, the event stream including the event data, wherein the remote capture agent generates the event data based on network data monitored by the remote capture agent;
execute the user-provided executable source code to transform the event data into transformed event data; and
send the transformed event data to another component for subsequent processing.
12. The configuration server of claim 11 , wherein the instructions, when executed by the processor, further cause the configuration server to send the transformed event data to a transformation server that performs at least one additional transformation to the transformed event data.
13. The configuration server of claim 11 , wherein the instructions, when executed by the processor, further cause the configuration server to:
generate configuration information based at least in part on the input configuring the transformation; and
send, to the remote capture agent, the configuration information, wherein the remote capture agent generates the event stream based on configuration information obtained by the remote capture agent from a configuration server.
14. The configuration server of claim 11 , wherein the remote capture agent generates the event stream based on configuration information obtained by the remote capture agent from the configuration server, and wherein the instructions, when executed by the processor, further cause the configuration server to:
receive an update to the configuration information to obtain updated configuration information; and
send the updated configuration information to the remote capture agent, wherein the remote capture agent uses the updated configuration information to generate additional event data.
15. The configuration server of claim 11 , wherein the instructions, when executed by the processor, further cause the configuration server to further transform the event data into the transformed event data by performing at least one of: an aggregation, a calculation, a filter, a normalization, or a formatting.
16. The configuration server of claim 11 , wherein the instructions, when executed by the processor, further cause to the configuration server to further transform the event data by:
obtaining a time interval associated with the event data; and
aggregating the event data within the time interval into at least one of: an event count, a statistic, or a uniqueness count.
17. The configuration server of claim 11 , wherein the remote capture agent generates the event data based on configuration information obtained by the remote capture agent from the configuration server, and wherein the configuration server obtains the configuration information from an application used to access the transformed event data.
18. The configuration server of claim 11 , wherein the user-provided executable source code is specified using an interpreted programming language.
19. The configuration server of claim 11 , wherein the remote capture agent is installed in a virtual computing environment.
20. The configuration server of claim 11 , wherein the user-provided executable source code causes the transformed event data to be stored at a storage location specified by the user-provided executable source code.
21. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause a configuration server coupled to a network to perform operations comprising:
receiving input configuring a transformation to be applied to event data of an event stream generated by a remote capture agent, wherein the input configuring the transformation includes user-provided executable source code that, upon execution, performs an operation to transform the event data into transformed event data;
receiving, from the remote capture agent, the event stream including the event data, wherein the remote capture agent generates the event data based on network data monitored by the remote capture agent;
executing the user-provided executable source code to transform the event data into transformed event data; and
sending the transformed event data to another component for subsequent processing.
22. The non-transitory computer-readable storage medium of claim 21 , wherein the instructions, when executed by the computer, further cause the remote capture agent to perform operations comprising sending the transformed event data to a transformation server that performs at least one additional transformation to the transformed event data.
23. The non-transitory computer-readable storage medium of claim 21 , wherein the instructions, when executed by the computer, further cause the configuration server to:
generate configuration information based at least in part on the input configuring the transformation; and
send, to the remote capture agent, the configuration information, wherein the remote capture agent generates the event stream based on configuration information obtained by the remote capture agent from a configuration server.
24. The non-transitory computer-readable storage medium of claim 21 , wherein the remote capture agent generates the event stream based on configuration information obtained by the remote capture agent from the configuration server, and wherein the instructions, when executed by the computer, further cause the configuration server to perform operations comprising:
receiving an update to the configuration information to obtain updated configuration information; and
sending the updated configuration information to the remote capture agent, wherein the remote capture agent uses the updated configuration information to generate additional event data.
25. The non-transitory computer-readable storage medium of claim 21 , wherein the instructions, when executed by the computer, further cause the configuration server to perform operations comprising further transforming the event data into the transformed event data by performing at least one of: an aggregation, a calculation, a filter, a normalization, or a formatting.
26. The non-transitory computer-readable storage medium of claim 21 , wherein the instructions, when executed by the computer, further cause the configuration server to perform operations comprising:
obtaining a time interval associated with the event data; and
aggregating the event data within the time interval into at least one of: an event count, a statistic, or a uniqueness count.
27. The non-transitory computer-readable storage medium of claim 21 , wherein the remote capture agent generates the event data based on configuration information obtained by the remote capture agent from the configuration server, and wherein the configuration server obtains the configuration information from an application used to access the transformed event data.
28. The non-transitory computer-readable storage medium of claim 21 , wherein the user-provided executable source code is specified using an interpreted programming language.
29. The non-transitory computer-readable storage medium of claim 21 , wherein the remote capture agent is installed in a virtual computing environment.
30. The non-transitory computer-readable storage medium of claim 21 , wherein the user-provided executable source code causes the transformed event data to be stored at a storage location specified by the user-provided executable source code.