Distributed processing of network data using remote capture agents
View Patent ↗The disclosed embodiments provide a method and system for processing network data. During operation, the system obtains one or more event streams from one or more remote capture agents over one or more networks, wherein the one or more event streams include event data generated from network packets captured by the one or more remote capture agents. Next, the system applies one or more transformations to the one or more event streams to obtain transformed event data from the event data. The system then enables querying of the transformed event data.
1. A computer-implemented method performed by a transformation server coupled via a network to a plurality of remote capture agents and used to improve processing of network data collected by the plurality of remote capture agents distributed across the network, the method comprising:
receiving configuration information from a configuration server over the network, wherein the configuration information is usable by the transformation server to generate event streams containing transformed timestamped events;
receiving timestamped events from one or more of the plurality of remote capture agents over the network, the timestamped events generated from network packets captured by the one or more of the plurality of remote capture agents;
identifying, in the configuration information received from the configuration server, information describing an event stream to be generated by transforming the timestamped events received from the one or more of the plurality of remote capture agents into transformed timestamped events, the information indicating one or more fields to be included in the transformed timestamped events and further indicating, for at least one field of the one or more fields, an identifier of data in the timestamped events to be transformed to obtain a value for the at least one field, and a type of transformation to apply to the data in the timestamped events to obtain the value for the at least one field; and
generating, based on the configuration information received from the configuration server, the event stream containing the transformed timestamped events by transforming the timestamped events received from the one or more of the plurality of remote capture agents into the transformed timestamped events.
2. The computer-implemented method of claim 1 , further comprising storing in a data store at least one of the timestamped events and the transformed timestamped events.
3. The computer-implemented method of claim 1 , further comprising enabling querying of the transformed timestamped events, wherein enabling querying of the transformed timestamped events comprises:
indexing the transformed timestamped events; and
executing queries on the indexed transformed timestamped events.
4. The computer-implemented method of claim 1 , further comprising:
indexing the transformed timestamped events; and
executing queries on the indexed transformed timestamped events, wherein execution of a query is performed on different subsets of the transformed timestamped events by one or more indexers in parallel.
5. The computer-implemented method of claim 1 , further comprising:
receiving a query requesting information from the transformed timestamped events; and
executing the query using at least one of a database and a log file.
6. The computer-implemented method of claim 1 , wherein the type of transformation comprises at least one of: an aggregation, a calculation, a filter, a normalization, and a formatting.
7. The computer-implemented method of claim 1 , wherein at least one of the timestamped events includes an identifier of a type of transaction associated with the at least one of the timestamped events.
8. The computer-implemented method of claim 1 , further comprising transmitting the transformed timestamped events over the network to a set of indexers, wherein the set of indexers are used to process queries using a late-binding schema of the transformed timestamped events.
9. The computer-implemented method of claim 1 , wherein the type of transformation is an aggregation operation that aggregates the timestamped events across multiple events to produce aggregated information, and generates transformed timestamped events including the aggregated information.
10. The computer-implemented method of claim 1 , wherein the information describing the event stream further indicates, for at least one field of the one or more fields, a name used to identify the at least one field.
11. A system used to improve processing of network data collected by a plurality of remote capture agents distributed across a network, comprising:
the plurality of remote capture agents implemented by a first one or more computing devices; and
a transformation server implemented by a second one or more computing devices, the transformation server including instructions that upon execution cause the transformation server to:
receive configuration information from a configuration server over the network, wherein the configuration information is usable by the transformation server to generate event streams containing transformed timestamped events;
receive timestamped events from one or more of the plurality of remote capture agents over the network, the timestamped events generated from network packets captured by the one or more of the plurality of remote capture agents;
identify, in the configuration information received from the configuration server, information describing an event stream to be generated by transforming the timestamped events received from the one or more of the plurality of remote capture agents into transformed timestamped events, the information indicating one or more fields to be included in the transformed timestamped events and further indicating, for at least one field of the one or more fields, an identifier of data in the timestamped events to be transformed to obtain a value for the at least one field, and a type of transformation to apply to the data in the timestamped events to obtain the value for the at least one field; and
generating, based on the configuration information received from the configuration server, the event stream containing the transformed timestamped events by transforming the timestamped events received from the one or more of the plurality of remote capture agents into the transformed timestamped events.
12. The system of claim 11 , wherein each of the plurality of remote capture agents includes instructions that upon execution cause the remote capture agent to:
generate timestamped events by applying one or more other transformations to the timestamped events; and
transmit the timestamped events to the one or more transformation servers.
13. The system of claim 11 , wherein the instructions, upon execution, further cause the transformation server to store in a data store at least one of: the timestamped events and the transformed timestamped events.
14. The system of claim 11 , wherein enabling the instructions, upon execution, further cause the transformation server to:
index the transformed timestamped events; and
execute queries on the indexed transformed timestamped events.
15. The system of claim 11 , wherein the instructions, upon execution, further cause the transformation server to:
index the transformed timestamped events; and
execute queries on the indexed transformed timestamped events, wherein execution of a query is performed on different subsets of the transformed timestamped events by one or more indexers in parallel.
16. The system of claim 11 , wherein the type of transformation comprises at least one of: an aggregation, a calculation, a filter, a normalization, and a formatting.
17. The system of claim 11 , further comprising an indexer implemented by a third one or more computing devices, the indexer including instructions that upon execution cause the indexer to:
receive the transformed timestamped events over the network from the transformation server; and
processing a query using a late-binding schema of the transformed timestamped events.
18. The system of claim 11 , wherein the type of transformation is an aggregation operation that aggregates the timestamped events across multiple events to produce aggregated information, and generates transformed timestamped events including the aggregated information.
19. The system of claim 11 , wherein the information describing the event stream further indicates, for at least one field of the one or more fields, a name used to identify the at least one field.
20. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause performance of operations comprising:
receiving configuration information from a configuration server over a network, wherein the configuration information is usable by a transformation server to generate event streams containing transformed timestamped events;
receiving timestamped events from one or more of the plurality of remote capture agents over the network, the timestamped events generated from network packets captured by the one or more of the plurality of remote capture agents;
identifying, in the configuration information received from the configuration server, information describing an event stream to be generated by transforming the timestamped events received from the one or more of the plurality of remote capture agents into transformed timestamped events, the information indicating one or more fields to be included in the transformed timestamped events and further indicating, for at least one field of the one or more fields, an identifier of data in the timestamped events to be transformed to obtain a value for the at least one field, and a type of transformation to apply to the data in the timestamped events to obtain the value for the at least one field; and
generating, based on the configuration information received from the configuration server, the event stream containing the transformed timestamped events by transforming the timestamped events received from the one or more of the plurality of remote capture agents into the transformed timestamped events.
21. The non-transitory computer-readable storage medium of claim 20 , wherein the instructions, when executed by the one or more processors, further cause performance of operations comprising storing in a data store at least one of: the timestamped events and the transformed events.
22. The non-transitory computer-readable storage medium of claim 20 , wherein the instructions, when executed by the one or more processors, further cause performance of operations comprising enabling querying of the transformed timestamped events, wherein enabling querying of the transformed timestamped events comprises:
indexing the transformed timestamped events; and
executing queries on the indexed transformed timestamped events.
23. The non-transitory computer-readable storage medium of claim 20 , wherein the instructions, when executed by the one or more processors, further cause performance of operations comprising:
indexing the transformed timestamped events; and
executing queries on the indexed transformed timestamped events, wherein execution of a query is performed on different subsets of the transformed timestamped events by one or more indexers in parallel.
24. The non-transitory computer-readable storage medium of claim 20 , wherein the type of transformation comprises at least one of: an aggregation, a calculation, a filter, a normalization, and a formatting.
25. The non-transitory computer-readable storage medium of claim 20 , wherein at least one of the timestamped events includes an identifier of a type of transaction associated with the at least one of the timestamped events.
26. A non-transitory computer-readable storage medium of claim 20 , wherein the type of transformation is an aggregation operation that aggregates the timestamped events across multiple events to produce aggregated information, and generates transformed timestamped events including the aggregated information.
27. A non-transitory computer-readable storage medium of claim 20 , wherein the information describing the event stream further indicates, for at least one field of the one or more fields, a name used to identify the at least one field.