Policy handling for data pipelines
Methods, systems, and devices for data processing are described. In some systems, data pipelines may be implemented to handle data processing jobs. To improve data pipeline flexibility, the systems may use separate pipeline and policy declarations. For example, a pipeline server may receive both a pipeline definition defining a first set of data operations to perform and a policy definition including instructions for performing a second set of data operations, where the first set of data operations is a subset of the second set. The server may execute a data pipeline based on a trigger (e.g., a scheduled trigger, a received message, etc.). To execute the pipeline, the server may layer the policy definition into the pipeline definition when creating an execution plan. The server may execute the execution plan by performing a number of jobs using a set of resources and plugins according to the policy definition.
1. A method for data pipeline execution at a server, comprising:
receiving, at the server, a pipeline definition comprising a first set of data operations to perform, the pipeline definition indicating a first network device for performing at least one data operation of the first set of data operations;
receiving, at the server, a policy definition comprising instructions for performing a second set of data operations, wherein the first set of data operations is a subset of the second set of data operations;
overriding the first network device indicated by the pipeline definition based at least in part on the policy definition;
identifying a second network device different from the first network device for performing the at least one data operation based at least in part on the pipeline definition and the policy definition;
generating an execution plan based at least in part on the identified second network device, the pipeline definition, and the policy definition; and
executing the execution plan, wherein the executing comprises performing the first set of data operations according to the instructions.
2. The method of claim 1 , further comprising:
determining a network device type indicated by the pipeline definition for performing the at least one data operation, wherein the second network device is identified based at least in part on the network device type indicated by the pipeline definition and the policy definition.
3. The method of claim 1 , wherein the executing comprises:
running a portion of code on the identified second network device based at least in part on the execution plan, wherein the portion of code performs the at least one data operation of the first set of data operations.
4. The method of claim 1 , wherein generating the execution plan comprises:
identifying a set of options indicated by the pipeline definition for performing the first set of data operations; and
selecting an option from the set of options based at least in part on the policy definition.
5. The method of claim 1 , further comprising:
receiving, at the server, an additional policy definition comprising additional instructions for performing a third set of data operations, wherein the first set of data operations is a subset of the third set of data operations;
regenerating the execution plan based at least in part on the pipeline definition and the additional policy definition; and
executing the regenerated execution plan, wherein the executing the regenerated execution plan comprises performing the first set of data operations according to the additional instructions.
6. The method of claim 1 , wherein:
the pipeline definition is received from a first user device; and
the policy definition is received from a second user device different from the first user device.
7. The method of claim 1 , wherein the pipeline definition comprises one or more triggers, the method further comprising:
identifying a trigger of the one or more triggers, wherein the execution plan is executed based at least in part on the identified trigger.
8. The method of claim 7 , wherein the execution plan is generated based at least in part on the identified trigger, the method further comprising:
transmitting a request for the policy definition based at least in part on the identified trigger, wherein the policy definition is received based at least in part on the request.
9. The method of claim 7 , wherein the identified trigger comprises a periodic trigger, a data-based trigger, or a combination thereof.
10. The method of claim 1 , wherein the instructions comprise a data type constraint and wherein the executing comprises:
determining whether data corresponds to a data type of the data type constraint; and
extracting the data if the data is determined to correspond to the data type of the data type constraint.
11. The method of claim 1 , wherein:
the instructions comprise a data storage constraint;
the generating the execution plan comprises identifying a data storage system corresponding to the data storage constraint; and
the executing comprises storing data in the data storage system based at least in part on the execution plan.
12. The method of claim 1 , wherein the instructions comprise a resource constraint for execution and wherein the executing comprises:
halting the executing based at least in part on a resource usage for the executing meeting or exceeding the resource constraint.
13. The method of claim 1 , wherein the executing comprises:
performing a plurality of data operations of the first set of data operations in sequence, in parallel, or in a combination thereof based at least in part on the pipeline definition.
14. The method of claim 1 , wherein the first set of data operations comprises monitoring for data, transforming data, processing data, extracting data, loading data, or a combination thereof.
15. An apparatus for data pipeline execution at a server, comprising:
a processor;
memory coupled with the processor; and
instructions stored in the memory and executable by the processor to cause the apparatus to:
receive, at the server, a pipeline definition comprising a first set of data operations to perform, the pipeline definition indicating a first network device for performing at least one data operation of the first set of data operations;
receive, at the server, a policy definition comprising policy instructions for performing a second set of data operations, wherein the first set of data operations is a subset of the second set of data operations;
override the first network device indicated by the pipeline definition based at least in part on the policy definition;
identify a second network device different from the first network device for performing the at least one data operation based at least in part on the pipeline definition and the policy definition;
generate an execution plan based at least in part on the identified second network device, the pipeline definition, and the policy definition; and
execute the execution plan, wherein the executing comprises performing the first set of data operations according to the policy instructions.
16. A non-transitory computer-readable medium storing code for data pipeline execution at a server, the code comprising instructions executable by a processor to:
receive, at the server, a pipeline definition comprising a first set of data operations to perform, the pipeline definition indicating a first network device for performing at least one data operation of the first set of data operations;
receive, at the server, a policy definition comprising policy instructions for performing a second set of data operations, wherein the first set of data operations is a subset of the second set of data operations;
override the first network device indicated by the pipeline definition based at least in part on the policy definition;
identify a second network device different from the first network device for performing the at least one data operation based at least in part on the pipeline definition and the policy definition;
generate an execution plan based at least in part on the identified second network device, the pipeline definition, and the policy definition; and
execute the execution plan, wherein the executing comprises performing the first set of data operations according to the policy instructions.