Dynamic data processor for streaming and batch queries
Operational machine components of an information technology (IT) or other microprocessor- or microcontroller-permeated environment generate disparate forms of machine data. Network connections are established between these components and processors of data intake and query system (DIQS). The DIQS conducts network transactions on a periodic and/or continuous basis with the machine components to receive disparate data and ingest certain of the data as entries of a data store that is searchable for DIQS query processing. The DIQS may receive queries to process against the received and ingested data via an exposed network interface. In one example embodiment, the DIQS receives a query identifying data to be processed, dynamically generates a query processing scheme based on the state of the data to be processed, such as streaming or at rest, and dynamically communicates the query processing scheme to a query executor based on the state of the data to be processed.
1. A method, comprising:
identifying, by a computing system, a first set of data to be processed;
determining a state of the first set of data is one of at rest or streaming;
generating, by the computing system, a first data processing scheme for obtaining and processing the first set of data based on the determined state of the first set of data;
selecting, based on the determined state of the first set of data, a first data processor from a plurality of data processors to execute the first data processing scheme;
communicating the first data processing scheme to the first data processor based on the determined state of the first set of data, wherein the first data processor executes the first data processing scheme;
identifying a second set of data to be processed;
determining a state of the second set of data is the other of at rest or streaming;
generating a second data processing scheme for obtaining and processing the second set of data based on the determined state of the second set of data;
selecting, based on the determined state of the second set of data, a second data processor from the plurality of data processors to execute the second data processing scheme; and
communicating the second data processing scheme to the second data processor based on the determined state of the second set of data, wherein the second data processor executes the second data processing scheme.
2. The method of claim 1 , wherein the first data processing scheme is a first query execution scheme, and generating the first query execution scheme comprises:
parsing a query,
generating a preliminary first data processing scheme based on the parsing, and
applying one or more optimizations to the preliminary first data processing scheme to generate the first query execution scheme.
3. The method of claim 1 , wherein the first data processing scheme is a modified first data processing scheme, and generating the modified first data processing scheme comprises:
parsing a query,
generating a preliminary first data processing scheme based on the parsing, and
generating the modified first data processing scheme based on an application of one or more streaming specific optimizations to the preliminary first data processing scheme.
4. The method of claim 1 , wherein the first data processing scheme is a modified first data processing scheme, and generating the modified first data processing scheme comprises:
parsing a query,
generating a preliminary first data processing scheme based on the parsing, and
generating the modified first data processing scheme based on an application of a set of streaming-specific configurations to the preliminary first data processing scheme, wherein the set of streaming-specific configurations are applied to the preliminary first data processing scheme based on the determining the state of the first set of data is streaming.
5. The method of claim 1 , wherein the first data processing scheme is a modified first data processing scheme, and generating the modified first data processing scheme comprises:
parsing a query,
generating a preliminary first data processing scheme based on the parsing, and
generating the modified first data processing scheme based on an application of one or more general optimizations and one or more streaming-specific optimizations to the preliminary first data processing scheme, wherein the one or more streaming-specific optimizations are applied based on the determining the state of the first set of data is streaming.
6. The method of claim 1 , wherein the first data processing scheme is a modified first data processing scheme, and generating the modified first data processing scheme comprises:
parsing a query,
generating a preliminary first data processing scheme based on the parsing, and
generating the modified first data processing scheme based on an application of one or more general optimizations and one or more streaming-specific optimizations to the preliminary first data processing scheme, wherein the one or more streaming-specific optimizations are applied based on the determining the state of the first set of data is streaming and the one or more general optimizations are applied regardless of the state of the first set of data.
7. The method of claim 1 , wherein the first data processing scheme is a modified first data processing scheme, and generating the modified first data processing scheme comprises:
parsing a query,
generating a preliminary first data processing scheme based on the parsing, and
generating the modified first data processing scheme based on an application of a time constraint to one or more commands of the preliminary first data processing scheme, wherein the time constraint is applied based on the determining the state of the first set of data is streaming.
8. The method of claim 1 , wherein the second data processing scheme is a modified second data processing scheme, and generating the modified second data processing scheme comprises:
parsing a query,
generating a preliminary second data processing scheme based on the parsing, and
generating the modified second data processing scheme based on pushing one or more commands of the preliminary second data processing scheme to an external data store, wherein the one or more commands are pushed to the external data store based on the determining the state of the second set of data is at rest.
9. A computing system, comprising:
a plurality of data processors including a first data processor and a second data processor, wherein the first data processor is configured to process data of a first type and the second data processor is configured to process data of a second type; and
one or more processing devices configured to:
identify a first set of data to be processed;
determine a state of the first set of data is one of at rest or streaming;
generate a first data processing scheme for obtaining and processing the first set of data based on the determined state of the first set of data;
select, based on the determined state of the first set of data, a first data processor from a plurality of data processors to execute the first data processing scheme;
communicate the first data processing scheme to the first data processor based on the determined state of the first set of data, wherein the first data processor executes the first data processing scheme;
identify a second set of data to be processed;
determine a state of the second set of data is the other of at rest or streaming;
generate a second data processing scheme for obtaining and processing the second set of data based on the determined state of the second set of data;
select, based on the determined state of the second set of data, a second data processor from the plurality of data processors to execute the second data processing scheme; and
communicate the second data processing scheme to the second data processor based on the determined state of the second set of data, wherein the second data processor executes the second data processing scheme.
10. The computing system of claim 9 , wherein the first data processing scheme is a first query execution scheme, and wherein to generate the first query execution scheme, the one or more processing devices are configured to:
parse a query,
generate a preliminary first data processing scheme based on the parsing, and
apply one or more optimizations to the preliminary first data processing scheme to generate the first query execution scheme.
11. The computing system of claim 9 , wherein the first data processing scheme is a modified first data processing scheme, and wherein to generate the modified first data processing scheme, the one or more processing devices are configured to:
parse a query,
generate a preliminary first data processing scheme based on the parsing, and
generate the modified first data processing scheme based on an application of one or more streaming specific optimizations to the preliminary first data processing scheme.
12. The computing system of claim 9 , wherein the first data processing scheme is a modified first data processing scheme and wherein to generate the modified first data processing scheme, the one or more processing devices are configured to:
parse a query,
generate a preliminary first data processing scheme based on the parsing, and
generate the modified first data processing scheme based on an application of a set of streaming-specific configurations to the preliminary first data processing scheme, wherein the set of streaming-specific configurations are applied to the preliminary first data processing scheme based on the determining the state of the first set of data is streaming.
13. The computing system of claim 9 , wherein the first data processing scheme is a modified first data processing scheme and wherein to generate the modified first data processing scheme, the one or more processing devices are configured to:
parse a query,
generate a preliminary first data processing scheme based on the parsing, and
generate the modified first data processing scheme based on an application of one or more general optimizations and one or more streaming-specific optimizations to the preliminary first data processing scheme, wherein the one or more streaming-specific optimizations are applied based on the determining the state of the first set of data is streaming.
14. Non-transitory computer-readable media comprising computer-executable instructions that, when executed by a computing system, cause the computing system to:
identify a first set of data to be processed;
determine a state of the first set of data is one of at rest or streaming;
generate a first data processing scheme for obtaining and processing the first set of data based on the determined state of the first set of data;
select, based on the determined state of the first set of data, a first data processor from a plurality of data processors to execute the first data processing scheme;
communicate the first data processing scheme to the first data processor based on the determined state of the first set of data, wherein the first data processor executes the first data processing scheme;
identify a second set of data to be processed;
determine a state of the second set of data is the other of at rest or streaming;
generate a second data processing scheme for obtaining and processing the second set of data based on the determined state of the second set of data;
select, based on the determined state of the second set of data, a second data processor from the plurality of data processors to execute the second data processing scheme; and
communicate the second data processing scheme to the second data processor based on the determined state of the second set of data, wherein the second data processor executes the second data processing scheme.
15. The non-transitory computer-readable media of claim 14 , wherein the first data processing scheme is a modified first data processing scheme, and wherein to generate the modified first data processing scheme, the computer-executable instructions further cause the computing system to:
parse a query,
generate a preliminary first data processing scheme based on the parsing, and
generate the modified first data processing scheme based on an application of one or more general optimizations and one or more streaming-specific optimizations to the preliminary first data processing scheme, wherein the one or more streaming-specific optimizations are applied based on the determining the state of the first set of data is streaming and the one or more general optimizations are applied regardless of the state of the first set of data.
16. The non-transitory computer-readable media of claim 14 , wherein the first data processing scheme is a modified first data processing scheme, and wherein to generate the modified first data processing scheme, the computer-executable instructions further cause the computing system to:
parse a query,
generate a preliminary first data processing scheme based on the parsing, and
generate the modified first data processing scheme based on an application of a time constraint to one or more commands of the preliminary first data processing scheme, wherein the time constraint is applied based on the determining the state of the first set of data is streaming.
17. The non-transitory computer-readable media of claim 14 , wherein the second data processing scheme is a modified second data processing scheme, and wherein to generate the modified second data processing scheme, the computer-executable instructions further cause the computing system to:
parse a query,
generate a preliminary second data processing scheme based on the parsing, and
generate the modified second data processing scheme based on pushing one or more commands of the preliminary second data processing scheme to an external data store, wherein the one or more commands are pushed to the external data store based on the determining the state of the second set of data is at rest.