EXECUTION ENGINE WRAPPER CONTAINER FOR A DATA ANALYTICS SYSTEM
The present patent application discloses a data analytics system with a split plane architecture that utilizes a workflow management module in a control plane for cloud-based computing. The system receives workflow files from client devices and stores those workflow files in a cloud storage associated with the user. The data plane, which executes the workflow file, retrieves the file directly from the cloud storage or through an API provided by the workflow management module. A workflow engine wrapper, containing a stager system and an execution system, prepares workflow files for execution and stores generated results in the cloud storage.
1 . A method comprising:
receiving, at an execution service of a data plane, a request to execute a workflow in a packaged workflow file;
creating an execution engine wrapper container in the data plane based on a container image, the execution engine wrapper container including at least a stager system, a worker system, and an execution engine;
retrieving, by the stager system of the execution engine wrapper container, the packaged workflow file;
unpackaging, by the stager system, components of the packaged workflow file to a local file system and providing a job request to the worker system;
responsive to initiation by the worker system, executing the workflow of the packaged workflow file by the execution engine;
receiving, at the worker system, results from the execution of the workflow by the execution engine; and
providing, by the worker system, the results of the execution to a control plane.
2 . The method of claim 1 , further comprising:
accessing a container image database including a plurality of container images;
identifying a version of an execution engine for executing the packaged workflow file for the request;
selecting the container image with a version of the execution engine matching the identified version for the request; and
executing the workflow of the packaged workflow file using the execution engine of the identified version.
3 . The method of claim 2 , wherein one or more container images are configured to run on a first operating system and one or more other container images are configured to run on a second operating system.
4 . The method of claim 1 , wherein the execution engine wrapper container includes a data connection manager, and the method further comprising:
receiving, at the data connection manager, a request from the execution engine for credentials to one or more data sources required for executing the workflow; and
providing the credentials to the execution engine.
5 . The method of claim 1 , further comprising:
receiving the packaged workflow file from a control plane of the data analytics system.
6 . The method of claim 1 , further comprising:
logging, by a logger system of the execution engine wrapper container, metadata on the execution of the workflow; and
publishing the metadata to a message bus for consumption by a service in a control plane.
7 . A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising:
receiving, at an execution service of a data plane, a request to execute a workflow in a packaged workflow file;
creating an execution engine wrapper container in the data plane based on a container image, the execution engine wrapper container including at least a stager system, a worker system, and an execution engine;
retrieving, by the stager system of the execution engine wrapper container, the packaged workflow file;
unpackaging, by the stager system, components of the packaged workflow file to a local file system and providing a job request to the worker system;
responsive to initiation by the worker system, executing the workflow of the packaged workflow file by the execution engine;
receiving, at the worker system, results from the execution of the workflow by the execution engine; and
providing, by the worker system, the results of the execution to a control plane.
8 . The computer-readable medium of claim 7 , wherein the operations further comprise:
accessing a container image database including a plurality of container images;
identifying a version of an execution engine for executing the packaged workflow file for the request;
selecting the container image with a version of the execution engine matching the identified version for the request; and
executing the workflow of the packaged workflow file using the execution engine of the identified version.
9 . The computer-readable medium of claim 8 , wherein one or more container images are configured to run on a first operating system and one or more other container images are configured to run on a second operating system.
10 . The computer-readable medium of claim 7 , wherein the execution engine wrapper container includes a data connection manager, and the operations further comprising:
receiving, at the data connection manager, a request from the execution engine for credentials to one or more data sources required for executing the workflow; and
providing the credentials to the execution engine.
11 . The computer-readable medium of claim 7 , wherein the operations further comprise:
receiving the packaged workflow file from a control plane of the data analytics system.
12 . The computer-readable medium of claim 7 , wherein the operations further comprise:
logging, by a logger system of the execution engine wrapper container, metadata on the execution of the workflow; and
publishing the metadata to a message bus for consumption by a service in a control plane.
13 . A system comprising:
a processor; and
a non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising:
receiving, at an execution service of a data plane, a request to execute a workflow in a packaged workflow file;
creating an execution engine wrapper container in the data plane based on a container image, the execution engine wrapper container including at least a stager system, a worker system, and an execution engine;
retrieving, by the stager system of the execution engine wrapper container, the packaged workflow file;
unpackaging, by the stager system, components of the packaged workflow file to a local file system and providing a job request to the worker system;
responsive to initiation by the worker system, executing the workflow of the packaged workflow file by the execution engine;
receiving, at the worker system, results from the execution of the workflow by the execution engine; and
providing, by the worker system, the results of the execution to a control plane.
14 . The system of claim 13 , wherein the operations further comprise:
accessing a container image database including a plurality of container images;
identifying a version of an execution engine for executing the packaged workflow file for the request;
selecting the container image with a version of the execution engine matching the identified version for the request; and
executing the workflow of the packaged workflow file using the execution engine of the identified version.
15 . The system of claim 14 , wherein one or more container images are configured to run on a first operating system and one or more other container images are configured to run on a second operating system.
16 . The system of claim 13 , wherein the execution engine wrapper container includes a data connection manager, and the operations further comprising:
receiving, at the data connection manager, a request from the execution engine for credentials to one or more data sources required for executing the workflow; and
providing the credentials to the execution engine.
17 . The system of claim 13 , wherein the operations further comprise:
receiving the packaged workflow file from a control plane of the data analytics system.
18 . The system of claim 13 , wherein the operations further comprise:
logging, by a logger system of the execution engine wrapper container, metadata on the execution of the workflow; and
publishing the metadata to a message bus for consumption by a service in a control plane.