Customizing operator nodes for graphical representations of data processing pipelines
A method may include receiving, from a client, a request to customize an operator node corresponding to a data processing operation. The request may include a first key. The operator node may be selected for inclusion in a graph representative of a data processing pipeline. The operator node may be associated with a first file that includes at least one configuration parameter associated with the operator node. The at least one configuration parameter may be associated with a second key. In response to the first key being determined to match the second key, the operator node may be customized by modifying the at least one configuration parameter. Furthermore, a second file associated with a customized operator node may be generated to store the customizations made to the operator node including the modification of the at least one configuration parameter. Related systems and articles of manufacture are also provided.
1. A system, comprising:
at least one data processor; and
at least one memory storing instructions which, when executed by the at least one data processor, result in operations comprising:
receiving, from a client, a request to customize an operator node corresponding to a data processing operation, the request including a first key and a first value, the operator node being selected for inclusion in a graph representative of a data processing pipeline, and the operator node associated with a plurality of configuration parameters, wherein the plurality of configuration parameters forms a tree structure;
determining, by traversing the tree structure, whether the first key matches a second key included in a first file associated with the operator node, the first file including a first configuration parameter associated with the operator node, the first configuration parameter associated with the second key and a second value;
in response to the first key being determined to match the second key, customizing the operator node by changing the second value associated with the first configuration parameter to the first value, the first value specified by the request, and generating a second file associated with a customized operator node, the second file storing one or more customizations made to the operator node in order to generate the customized operator node, and the one or more customizations including a modification to the first configuration parameter;
adding, to the graph representative of the data processing pipeline, the customized operator node, the customized operator node corresponding to a customized data processing operation, and the addition of the customized operator node to the graph comprising an addition of the customized data processing operation to a series of data processing operations included in the data processing pipeline; and
generating, based at least on the graph, the corresponding data processing pipeline, the data processing pipeline including the customized data processing operation corresponding to the customized operator node.
2. The system of claim 1 , further comprising:
in response to the first key being determined not to match the second key,
customizing the operator node by at least adding, to the operator node, a second configuration parameter, the second configuration parameter being associated with the first key, and
generating the second file to include the addition of the second configuration parameter.
3. The system of claim 1 , wherein the first file includes a plurality of key-value pairs corresponding to the plurality of configuration parameters associated with the operator node.
4. The system of claim 1 , wherein the one or more customization are stored in the second file associated with the customized operator node instead of the first file associated with the operator node.
5. The system of claim 1 , further comprising:
executing the data processing pipeline by at least applying, to the data stored in a database, the series of data processing operations including the customized data processing operation corresponding to the customized operator node.
6. The system of claim 1 , wherein the first file and/or the second file comprise a JavaScript Object Notation (JSON) file.
7. A computer-implemented method, comprising:
receiving, from a client, a request to customize an operator node corresponding to a data processing operation, the request including a first key and a first value, the operator node being selected for inclusion in a graph representative of a data processing pipeline, and the operator node associated with a plurality of configuration parameters, wherein the plurality of configuration parameters forms a tree structure;
determining, by traversing the tree structure, whether the first key matches a second key included in a first file associated with the operator node, the first file including a first configuration parameter associated with the operator node, the first configuration parameter associated with the second key and a second value;
in response to the first key being determined to match the second key, customizing the operator node by changing the second value associated with the first configuration parameter to the first value, the first value specified by the request, and generating a second file associated with a customized operator node, the second file storing one or more customizations made to the operator node in order to generate the customized operator node, and the one or more customizations including the modification to the first configuration parameter;
adding, to the graph representative of the data processing pipeline, the customized operator node, the customized operator node corresponding to a customized data processing operation, and the addition of the customized operator node to the graph comprising an addition of the customized data processing operation to a series of data processing operations included in the data processing pipeline; and
generating, based at least on the graph, the corresponding data processing pipeline, the data processing pipeline including the customized data processing operation corresponding to the customized operator node.
8. The method of claim 7 , further comprising:
in response to the first key being determined not to match the second key,
customizing the operator node by at least adding, to the operator node, a second configuration parameter, the second configuration parameter being associated with the first key, and
generating the second file to include the addition of the second configuration parameter.
9. The method of claim 7 , wherein the first file includes a plurality of key-value pairs corresponding to the plurality of configuration parameters associated with the operator node.
10. The method of claim 7 , wherein the one or more customization are stored in the second file associated with the customized operator node instead of the first file associated with the operator node.
11. The method of claim 7 , further comprising:
executing the data processing pipeline by at least applying, to the data stored in a database, the series of data processing operations including the customized data processing operation corresponding to the customized operator node.
12. A non-transitory computer-readable medium storing instructions, which when executed by at least one data processor, result in operations comprising:
receiving, from a client, a request to customize an operator node corresponding to a data processing operation, the request including a first key and a first value, the operator node being selected for inclusion in a graph representative of a data processing pipeline, and the operator node associated with a plurality of configuration parameters, wherein the plurality of configuration parameters forms a tree structure;
determining, by traversing the tree structure, whether the first key matches a second key included in a first file associated with the operator node, the first file including a first configuration parameter associated with the operator node, the first configuration parameter associated with the second key and a second value;
in response to the first key being determined to match the second key, customizing the operator node by changing the second value associated with the first configuration parameter to the first value, the first value specified by the request, and generating a second file associated with a customized operator node, the second file storing one or more customizations made to the operator node in order to generate the customized operator node, and the one or more customizations including the modification to the first configuration parameter;
adding, to the graph representative of the data processing pipeline, the customized operator node, the customized operator node corresponding to a customized data processing operation, and the addition of the customized operator node to the graph comprising an addition of the customized data processing operation to a series of data processing operations included in the data processing pipeline; and
generating, based at least on the graph, the corresponding data processing pipeline, the data processing pipeline including the customized data processing operation corresponding to the customized operator node.