WORKFLOW FILE MANAGEMENT BY A CLOUD-BASED DATA ANALYTICS SYSTEM
A data analytics system that includes a control plane for executing workflows is described. The data analytics system receives workflow files from client devices and stores the workflow files in respective storage accounts associated with the respective client device profiles. A virtualized container separate from the control plane executes the workflow file by retrieving the workflow file directly from a storage account or via an application programming interface provided by the control plane. Results generated by the virtualized container are stored in a corresponding storage account for subsequent access, such as by a client device from which a workflow execution request was received.
1 . A method comprising:
receiving, at a control plane of a data analytics system, a packaged workflow file describing at least one workflow to be executed by the data analytics system;
storing, by the control plane, the packaged workflow file to a storage account of the data analytics system, the storage account associated with a client device;
receiving, by the control plane and from the client device, a request to execute a job;
causing, by the control plane and in response to receiving the request, a virtualized container to execute the job by publishing a workflow job message to a message queue, the workflow job message causing the virtualized container to execute the job using the packaged workflow file;
receiving, by the control plane, a result of the virtualized container executing the job using the packaged workflow file; and
outputting, by the control plane, the result to the storage account.
2 . The method of claim 1 , further comprising:
extracting a workflow from the packaged workflow file; and
converting the extracted workflow to a different format and storing the different format of the extracted workflow in a workflow database accessible by the control plane.
3 . The method of claim 2 , wherein the extracted workflow is represented using Extensible Markup Language (XML) and the different format of the extracted workflow is represented using JavaScript Object Notation (JSON).
4 . The method of claim 1 , further comprising outputting a user interface for display at the client device, the user interface including a representation of the at least one workflow stored in the storage account, wherein receiving the request to execute the job is performed responsive to a selection, at the client device, of the representation of the at least one workflow.
5 . The method of claim 1 , wherein the packaged workflow file comprises one or more dependency files for the at least one workflow, the one or more dependency files including at least one of an executable, an embedded dataset, an image, a macro, a database file, or a plugin.
6 . The method of claim 1 , wherein the virtualized container is executed by a computing system that is different than the client device and different than one or more processing devices executing the control plane of the data analytics system.
7 . The method of claim 6 , further comprising:
receiving, by the control plane and in response to publishing the workflow job message to the message queue, a request from the computing system executing the virtualized container to access the packaged workflow file;
authenticating, by the control plane, the computing system executing the virtualized container; and
authorizing the virtualized container to access the packaged workflow file responsive to authenticating the computing system executing the virtualized container.
8 . The method of claim 1 , wherein the virtualized container comprises:
a staging system configured to retrieve and unpack the packaged workflow file;
a worker system configured to manage execution of the at least one workflow; and
an execution engine configured to perform operations specified by the at least one workflow.
9 . The method of claim 1 , further comprising:
monitoring, by the control plane, execution progress of the job by the virtualized container;
detecting an error condition during execution of the job by the virtualized container; and
initiating error handling based on the error condition.
10 . The method of claim 1 , further comprising:
receiving, by the control plane, a request to modify the packaged workflow file;
creating a new version of the packaged workflow file based on the request to modify the packaged workflow file;
storing the new version of the packaged workflow file in the storage account; and
maintaining version control information for the packaged workflow file, the version control information describing the packaged workflow file and the new version of the packaged workflow file.
11 . A data analytics system comprising:
at least one processor; and
a computer-readable storage medium storing instructions that are executable by the at least one processor to perform operations comprising:
instantiating a control plane of the data analytics system;
receiving, at the control plane, a packaged workflow file describing at least one workflow to be executed;
storing, by the control plane, the packaged workflow file at a storage account associated with a client device;
receiving, by the control plane and from the client device, a request to execute a job;
causing, by the control plane and in response to receiving the request, a virtualized container to execute the job by publishing a workflow job message to a message queue, the workflow job message causing the virtualized container to execute the job using the packaged workflow file;
receiving, by the control plane, a result of the virtualized container executing the job using the packaged workflow file; and
outputting, by the control plane, the result to the storage account.
12 . The data analytics system of claim 11 , the operations further comprising:
extracting a workflow from the packaged workflow file; and
converting the extracted workflow to a different format and storing the different format of the extracted workflow in a workflow database accessible by the control plane.
13 . The data analytics system of claim 12 , wherein the extracted workflow is represented using Extensible Markup Language (XML) and the different format of the extracted workflow is represented using JavaScript Object Notation (JSON).
14 . The data analytics system of claim 11 , the operations further comprising outputting a user interface for display at the client device, the user interface including a representation of the at least one workflow stored in the storage account, wherein receiving the request to execute the job is performed responsive to a selection, at the client device, of the representation of the at least one workflow.
15 . The data analytics system of claim 11 , wherein the packaged workflow file comprises one or more dependency files for the at least one workflow, the one or more dependency files including at least one of an executable, an embedded dataset, an image, a macro, a database file, or a plugin.
16 . The data analytics system of claim 11 , wherein the virtualized container is executed by a computing device that is different than the client device and different than the data analytics system.
17 . The data analytics system of claim 16 , the operations further comprising:
receiving, by the control plane and in response to publishing the workflow job message to the message queue, a request from the computing device executing the virtualized container to access the packaged workflow file;
authenticating, by the control plane, the computing device executing the virtualized container; and
authorizing the virtualized container to access the packaged workflow file responsive to authenticating the computing device executing the virtualized container.
18 . The data analytics system of claim 11 , wherein the virtualized container comprises:
a staging system configured to retrieve and unpack the packaged workflow file;
a worker system configured to manage execution of the at least one workflow; and
an execution engine configured to perform operations specified by the at least one workflow.
19 . The data analytics system of claim 11 , the operations further comprising:
monitoring, by the control plane, execution progress of the job by the virtualized container;
detecting an error condition during execution of the job by the virtualized container; and
initiating error handling based on the error condition.
20 . A non-transitory computer-readable storage medium storing instructions that are executable by at least one processing device to perform operations comprising:
instantiating a control plane of a data analytics system;
receiving, at the control plane, a packaged workflow file describing at least one workflow to be executed;
storing, by the control plane, the packaged workflow file at a storage account associated with a client device;
receiving, by the control plane and from the client device, a request to execute a job;
causing, by the control plane and in response to receiving the request, a virtualized container to execute the job by publishing a workflow job message to a message queue, the workflow job message causing the virtualized container to execute the job using the packaged workflow file;
receiving, by the control plane, a result of the virtualized container executing the job using the packaged workflow file; and
outputting, by the control plane, the result to the storage account