System and method facilitating reusability of distributed computing pipelines
A system and method for facilitating reusability of distributed computing pipelines, initially, captures the distributed computing pipeline designed over a Graphical User Interface (GUI) of a first data processing environment associated with a stream analytics platform. Subsequent to the designing, the distributed computing pipeline may be stored in a repository. The distributed computing pipeline may be stored in a file with a predefined file format pertaining to the stream analytics platform. The system also maintains a repository of different versions of the distributed computing pipeline created by the first and second user. Upon storing the file, the file may be imported in a second data processing environment. After importing the file, the distributed computing pipeline may be populated over the GUI of the second data processing environment, thereby facilitating reusability of the distributed computing pipeline.
1. A method for facilitating reusability of distributed computing pipelines, the method comprising:
capturing, by a processor, a set of versions pertaining to a distributed computing pipeline designed by a first user, over a Graphical User Interface (GUI), of a first data processing environment associated with a stream analytics platform, wherein each version indicates same or distinct business logic, and wherein the distributed computing pipeline comprises:
a subset of components, selected by the first user from a universal set of components of the stream analytics platform, and
a set of links corresponding to the subset of components;
storing, by the processor, the distributed computing pipeline with the set of versions in a repository, wherein the distributed computing pipeline with the set of versions are stored in a file with a predefined file format pertaining to the stream analytics platform, wherein each version of pipeline is stored along with metadata associated with each version of the pipeline, wherein the file is configured to maintain components and configurations of the distributed computing pipeline comprising a name, a structure, messages, message groups, alerts, agent configurations, transformation variables, scope variables, wherein user defined functions and registered components enables the user for designing a new pipeline or modifying an existing pipeline, wherein the registered components have list of all the custom components registered by user;
importing or exporting, by the processor, the file in a second data processing environment, wherein exporting or importing of the pipelines is done amongst different data processing environment by using at least one communication medium;
verifying, by the processor, compatibility of the file in the second data processing environment, wherein the compatibility is verified by ensuring the predefined file format is conflict free for the second data processing environment;
updating, by the processor, the file as per the second data processing environment when the file is in conflict with the second data processing environment; and
populating, by the processor, a version of the distributed computing pipeline over the GUI of the second data processing environment, thereby facilitating the second user of the second data processing environment to reuse the distributed computing pipeline.
2. The method of claim 1 further comprises updating, by the processor, one or more components of the distributed computing pipeline to generate one or more versions corresponding to the distributed computing pipeline.
3. The method of claim 1 , wherein the GUI comprises at least a canvas and a palette, wherein the palette is configured to display the universal set of components, and wherein the canvas enables the first user and the second user to drag and drop one or more components from the universal set of components for generating the distributed computing pipeline.
4. The method of claim 1 , wherein the subset of components comprises of at least a channel component, a processor component, an enricher component and an emitter component.
5. A system for facilitating reusability of distributed computing pipelines, the system comprising:
a processor; and
a memory coupled to the processor, wherein the processor is capable of executing a plurality of modules stored in the memory, and wherein the plurality of modules comprising:
a pipeline designing module is configured for capturing a set of versions pertaining to a distributed computing pipeline designed by a first user, over a Graphical User Interface (GUI) of a first data processing environment associated with a stream analytics platform, wherein each version indicates same or distinct business logic, and wherein the distributed computing pipeline comprises:
a subset of components, selected by the first user of the data processing environment from a universal set of components of the stream analytics platform, and a set of links corresponding to the subset of components;
an export module is configured for storing the distributed computing pipeline with the set of versions in a repository, wherein the distributed computing pipeline with the set of versions are stored in a file with a predefined file format pertaining to the stream analytics platform, wherein each version of pipeline is stored along with metadata associated with each version of the pipeline wherein the file is configured to maintain components and configurations of the distributed computing pipeline comprising a name, a structure, messages, message groups, alerts, agent configurations, transformation variables, scope variables, wherein user defined functions and registered components enables the user for designing a new pipeline or modifying an existing pipeline, wherein the registered components have list of all the custom components registered by user;
an import module is configured for importing the file in a second data processing environment, wherein exporting or importing of the pipelines is done amongst different data processing environment;
verifying compatibility of the file in the second data processing environment, wherein the compatibility is verified by ensuring the predefined file format is conflict free for the second data processing environment;
updating the file as per the second data processing environment when the file is in conflict with the second data processing environment; and
a populating module is configured for populating a version of the distributed computing pipeline over the GUI of the second data processing environment, thereby facilitating the second user of the second data processing environment to reuse the distributed computing pipeline.
6. The system of claim 5 further configured to update one or more components of the distributed computing pipeline to generate one or more versions corresponding to the distributed computing pipeline.
7. The system of claim 5 , wherein GUI comprises at least a canvas and a palette, wherein the palette is configured to display the universal set of components, and wherein the canvas enables the first user and the second user to drag and drop one or more components from the universal set of components for generating the distributed computing pipeline.
8. The system of claim 5 , wherein the subset of components comprises of at least a channel component, a processor component, an enricher component and an emitter component.
9. A non-transitory computer readable medium embodying a program executable in a computing device for facilitating reusability of distributed computing pipelines, the program comprising a program code:
a program code for capturing a set of versions pertaining to a distributed computing pipeline designed by a first user, over a Graphical User Interface (GUI), of a first data processing environment associated with a stream analytics platform, wherein each version indicates same or distinct business logic, and wherein the distributed computing pipeline comprises:
a subset of components, selected by the first user of the data processing environment from a universal set of components of the stream analytics platform, and
a set of links corresponding to the subset of components;
a program code for storing the distributed computing pipeline with the set of versions in a repository, wherein the distributed computing pipeline with the set of versions are stored in a file with a predefined file format pertaining to the stream analytics platform, wherein each version of pipeline is stored along with metadata associated with each version of the pipeline wherein the file is configured to maintain components and configurations of the distributed computing pipeline comprising a name, a structure, messages, message groups, alerts, agent configurations, transformation variables, scope variables, wherein user defined functions and registered components enables the user for designing a new pipeline or modifying an existing pipeline, wherein the registered components have list of all the custom components registered by user;
a program code for importing the file in a second data processing environment, wherein exporting or importing of the pipelines is done amongst different data processing environment;
a program code for verifying compatibility of the file in the second data processing environment, wherein the compatibility is verified by ensuring the predefined file format is conflict free for the second data processing environment;
a program code for updating the file as per the second data processing environment when the file is in conflict with the second data processing environment; and
a program code for populating a version of the distributed computing pipeline over the GUI of the second data processing environment, thereby facilitating the second user of the second data processing environment to reuse the distributed computing pipeline.
10. The method of claim 1 further comprises displaying warning message when the file is in conflict with the second data processing environment.
11. The system of claim 5 further comprises displaying warning message when the file is in conflict with the second data processing environment.
12. The method of claim 1 , wherein the first data processing environment and the second data processing environment work independently and individually, and wherein the first data processing environment or the second data processing environment is one of a pre-production environment, a production environment, a test environment, and a development environment, and wherein the first data processing environment and the second data processing environment are either in same network or distinct network.
13. The system of claim 5 , wherein the first data processing environment and the second data processing environment work independently and individually, and wherein the first data processing environment or the second data processing environment is one of a pre-production environment, a production environment, a test environment, and a development environment, and wherein the first data processing environment and the second data processing environment are either in same network or distinct network.
14. The method of claim 1 , wherein each version of the set of versions is having one or more folders created in the system database capable of storing the pipeline and metadata pertaining to each version of the distributed computing pipeline, wherein each folder is identified by a unique name and a unique version number, and wherein each folder stores pipeline definition in a predefined file format.
15. The system of claim 5 , wherein each version set of versions is having one or more folders created in the system database capable of storing the business logic and metadata pertaining to the distributed computing pipeline, wherein each folder is identified by a unique name and a unique version number, and wherein each folder stores pipeline definition in a predefined file format.