System and method for creation and implementation of data processing workflows using a distributed computational graph
A system and method for creating and implementing data processing workflows using a distributed computational graph comprising modules that represent various stages within a data processing workflow. Each module represents one or more data processing steps, with some of the modules representing data processing performed by a cloud-based service and containing code for interfacing with the application programming interface (API) of that cloud-based service. A series of modules and their interconnections specify the workflow. Data is processed according to the workflow by implementing the data processing step represented by each module, some of which may access cloud-based data processing services. The result is that users can create complex data processing workflows that utilize cloud-based services to process data without having to know how to access the cloud-based data processing services, or even know that they exist.
1. A system for creating and implementing data processing workflows using a distributed computational graph comprising:
a computing device comprising a memory, a processor, and a non-volatile data storage device;
a front end comprising a first plurality of programming instructions stored in the memory of, and operating on the processor of, the computing device, wherein the first plurality of programming instructions, when operating on the processor, cause the computing device to:
provide a graphical user interface for creation of a distributed computational graph representing a data processing workflow, wherein:
the graphical user interface comprises icons representing data processing modules and connections between the modules;
the distributed computational graph comprises one or more of the data processing modules represented as nodes of the distributed computational graph and connections between the data processing modules represented as edges of the distributed computational graph; and
one or more of the nodes of the distributed computational graph is a data processing module that contains code for accessing an application programing interface (API) of a cloud-based data processing service;
send the distributed computational graph to a system back end for implementation of the data processing workflow; and
a system back end comprising a second plurality of programming instructions stored in the memory of, and operating on the processor of, the computing device, wherein the second plurality of programming instructions, when operating on the processor, cause the computing device to:
receive the distributed computational graph;
construct the data processing workflow from the distributed computational graph using the application programing interface that generates workflow code via representational state transfer calls;
instantiate and validate the data processing workflow using the workflow code; prepare the data processing workflow for processing by initializing one or more of: software components, devices, cloud-based services;
serialize the data processing workflow as a series of processing events;
execute the processing events using a data processing engine.
2. A method for creating data processing workflows using modules, comprising the steps of:
providing a graphical user interface for creation of a distributed computational graph representing a data processing workflow, wherein:
the graphical user interface comprises icons representing data processing modules and connections between the modules;
the distributed computational graph comprises one or more of the data processing modules represented as nodes of the distributed computational graph and connections between the data processing modules represented as edges of the distributed computational graph; and
one or more of the nodes of the distributed computational graph is a data processing module that contains code for accessing an application programing interface (API) of a cloud-based data processing service;
constructing the data processing workflow from the distributed computational graph using the application programing interface that generates workflow code via representational state transfer calls;
instantiating and validating the data processing workflow using the workflow code;
preparing the data processing workflow for processing by initializing one or more of: software components, devices, cloud-based services;
serializing the data processing workflow as a series of processing events; and
executing the processing events using a data processing engine.