Configuration-based development of analytics pipelines
This disclosure describes techniques relating to development of data flow pipelines. In one example, this disclosure describes a method that includes accessing, by a computing system, metadata identifying characteristics of a data flow pipeline; generating, by the computing system and based on the metadata, the data flow pipeline, wherein the data flow pipeline includes a plurality of stages, and wherein the plurality of stages includes a multi-sourced stage in which data output by each of a subset of stages in the plurality of stages are used as input to the multi-sourced stage; executing, by the computing system, the data flow pipeline; detecting, by the computing system, modifications to the metadata; generating, based on the modifications, an updated data flow pipeline; and executing, by the computing system, the updated data flow pipeline.
1. A method comprising:
accessing, by a computing system, metadata identifying characteristics of a data flow pipeline, wherein the metadata includes a linear list of descriptions of a plurality of stages in the data flow pipeline, and wherein each of the descriptions in the linear list of descriptions is ordered in the metadata pursuant to a set of ordering rules;
constructing, by the computing system and based on the metadata, the data flow pipeline, wherein the data flow pipeline includes a plurality of stages, wherein the plurality of stages includes at least one single sourced stage and a multi-sourced stage in which data output by each of a subset of stages in the plurality of stages are used as input to the multi-sourced stage, and wherein constructing the data flow pipeline includes interpreting the linear list of descriptions in the metadata pursuant to the ordering rules so that (1) the description of each single-sourced stage receiving a single input in the pipeline is immediately preceded by a description of the stage that produces the single input, and (2) the description of the multi-sourced stage is preceded by a description, in a relative order, of each of the subset of stages that produce an input to the multi-sourced stage, wherein the relative order corresponds to an order in which each of the inputs to the multi-sourced stage is received by the multi-sourced stage;
monitoring, by the computing system, changes to the metadata, wherein monitoring the changes to the metadata includes monitoring a storage device storing the metadata;
processing, by the computing system, a first set of data using the data flow pipeline;
detecting, by the computing system, modifications to the metadata resulting in updated metadata, wherein detecting modifications to the metadata includes detecting addition of a reference to a first component in the metadata and detecting removal of a reference to a second component in the metadata;
constructing, by the computing system and based on the updated metadata, an updated data flow pipeline, wherein constructing the updated data flow pipeline includes interpreting the updated metadata by applying the ordering rules to the updated metadata to:
add a stage to the pipeline corresponding to the addition of the reference to the first component, and
remove a stage from the pipeline corresponding to the removal of the reference to the second component; and
processing, by the computing system, a second set of data using the updated data flow pipeline.
2. The method of claim 1 , wherein constructing the data flow pipeline includes:
interpreting the metadata to identify components included within the pipeline.
3. The method of claim 2 , wherein constructing the data flow pipeline includes:
interpreting the metadata to identify arguments associated with each of the identified components.
4. The method of claim 3 , wherein interpreting the metadata includes:
interpreting high-level descriptions of the components in the metadata, wherein the high-level descriptions provide information about operations performed by each respective component without requiring knowledge of source code used to implement the component.
5. The method of claim 1 , wherein constructing the data flow pipeline includes:
generating a logical data flow in preparation for processing the pipeline.
6. The method of claim 1 , wherein constructing the data flow pipeline and constructing the updated data flow pipeline both include:
executing a standardized driver.
7. A system comprising a storage system and processing circuitry having access to the storage system, wherein the processing circuitry is configured to:
access metadata identifying characteristics of a data flow pipeline, wherein the metadata includes a linear list of descriptions of a plurality of stages in the data flow pipeline, and wherein each of the descriptions in the linear list of descriptions is ordered in the metadata pursuant to a set of ordering rules;
construct, based on the metadata, the data flow pipeline, wherein the data flow pipeline includes a plurality of stages, wherein the plurality of stages includes at least one single sourced stage and a multi-sourced stage in which data output by each of a subset of stages in the plurality of stages are used as input to the multi-sourced stage, and wherein to construct the data flow pipeline, the processing circuitry is further configured to interpret the linear list of descriptions in the metadata pursuant to the ordering rules so that (1) the description of each single-sourced stage receiving a single input in the pipeline is immediately preceded by a description of the stage that produces the single input, and (2) the description of the multi-sourced stage is preceded by a description, in a relative order, of each of the subset of stages that produce an input to the multi-sourced stage, wherein the relative order corresponds to an order in which each of the inputs to the multi-sourced stage is received by the multi-sourced stage;
monitor changes to the metadata, wherein to monitor changes, the processing circuitry is configured to monitor a storage device storing the metadata;
process a first set of data using the data flow pipeline;
detect modifications to the metadata resulting in updated metadata, wherein to detect modifications to the metadata, the processing circuitry is configured to detect addition of a reference to a first component in the metadata and detect removal of a reference to a second component in the metadata;
construct, based on the updated metadata, an updated data flow pipeline, wherein to construct the updated data flow pipeline, the processing circuitry is configured to interpret the updated metadata by applying the ordering rules to the updated metadata to:
add a stage to the pipeline corresponding to the addition of the reference to the first component, and
remove a stage from the pipeline corresponding to the removal of the reference to the second component; and
process a second set of data using the updated data flow pipeline.
8. The system of claim 7 , wherein to construct the data flow pipeline, the processing circuitry is further configured to:
interpret the metadata to identify components included within the pipeline.
9. The system of claim 8 , wherein to construct the data flow pipeline, the processing circuitry is further configured to:
interpret the metadata to identify arguments associated with each of the identified components.
10. The system of claim 9 , wherein to interpret the metadata, the processing circuitry is further configured to:
interpret high-level descriptions of the components in the metadata, wherein the high-level descriptions provide information about operations performed by each respective component without requiring knowledge of source code used to implement the component.
11. The system of claim 7 , wherein to construct the data flow pipeline, the processing circuitry is further configured to:
generate a logical data flow in preparation for processing the pipeline.
12. A non-transitory computer-readable storage medium comprising instructions that, when executed, configure processing circuitry of a computing system to:
access metadata identifying characteristics of a data flow pipeline, wherein the metadata includes a linear list of descriptions of a plurality of stages in the data flow pipeline, and wherein each of the descriptions in the linear list of descriptions is ordered in the metadata pursuant to a set of ordering rules;
construct, based on the metadata, the data flow pipeline, wherein the data flow pipeline includes a plurality of stages, wherein the plurality of stages includes at least one single sourced stage and a multi-sourced stage in which data output by each of a subset of stages in the plurality of stages are used as input to the multi-sourced stage, and wherein to construct the data flow pipeline, the processing circuitry is further configured to interpret the linear list of descriptions in the metadata pursuant to the ordering rules so that (1) the description of each single-sourced stage receiving a single input in the pipeline is immediately preceded by a description of the stage that produces the single input, and (2) the description of the multi-sourced stage is preceded by a description, in a relative order, of each of the subset of stages that produce an input to the multi-sourced stage, wherein the relative order corresponds to an order in which each of the inputs to the multi-sourced stage is received by the multi-sourced stage;
monitor changes to the metadata, wherein to monitor changes, the processing circuitry is configured to monitor a storage device storing the metadata;
process a first set of data using the data flow pipeline;
detect modifications to the metadata resulting in updated metadata, wherein to detect modifications to the metadata, the processing circuitry is configured to detect addition of a reference to a first component in the metadata and detect removal of a reference to a second component in the metadata;
construct, based on the updated metadata, an updated data flow pipeline, wherein to construct the updated data flow pipeline, the processing circuitry is configured to interpret the updated metadata by applying the ordering rules to the updated metadata to:
add a stage to the pipeline corresponding to the addition of the reference to the first component, and
remove a stage from the pipeline corresponding to the removal of the reference to the second component; and
process a second set of data using the updated data flow pipeline.