Dynamically parallelized workflow processing
System and methods are provided for dynamically parallelized large data set processing. A workflow includes a distributed map state. A set of steps in the distributed map state are run for each item in a collection of items or for each batch. Execution of the distributed map state is dynamically parallelized such that the system creates as many parallel workflows as there are items in the collection at run time (or batches). The items in the collection are dynamically determined by the workflow system, as defined by a resource in the distributed map state.
1 . A system comprising:
a data store storing computer-executable instructions; and
one or more computer hardware processors in communication with the data store, wherein the one or more computer hardware processors are configured to execute the computer-executable instructions to at least:
receive a first workflow comprising a distributed map state, the distributed map state comprising a resource, a batch parameter, and a second workflow; and
execute the distributed map state, wherein to execute the distributed map state, the one or more computer hardware processors are configured to execute the computer-executable instructions to at least:
for each batch of a plurality of batches,
invoke, on a host computing device, a child execution of the second workflow, with the batch as input to the second workflow, wherein the plurality of batches is based at least in part on a plurality of items and the batch parameter, wherein the plurality of items is based at least in part on the resource,
and wherein completion of the child execution of the second workflow on the host computing device generates output; and
receive the output generated by the child execution of the second workflow on the host computing device; and
store the output from each batch of the plurality of batches.
2 . The system of claim 1 , wherein the one or more computer hardware processors are configured to execute further computer-executable instructions to at least:
cause presentation of a graphical user interface;
receive, via the graphical user interface, user input of a step in a visual workflow; and
generate the first workflow based at least in part on the step in the visual workflow.
3 . The system of claim 1 , wherein the second workflow comprises a first state and a second state, wherein output from the first state as input to the second state is limited to a first data input size, and wherein a data size of a collection of identifiers for the plurality of items exceeds the first data input size.
4 . The system of claim 1 ,
wherein each identifier of a collection of identifiers identifies an item from the plurality of items.
5 . The system of claim 1 , wherein the batch parameter is a first number,
wherein a count of items in each group of a plurality of groups of items, from the plurality of items, satisfies a threshold value,
wherein the threshold value is based at least in part on the first number, and
wherein each group from the plurality of groups of items is a batch.
6 . The system of claim 5 , wherein the first number is one.
7 . A computer-implemented method comprising:
receiving a first workflow definition defining a first workflow process that includes a distributed map state, the first workflow definition indicating, for the distributed map state, a resource, a batch parameter, and a second workflow;
executing the first workflow process in accordance with the first workflow definition; and
executing the distributed map state as part of the first workflow process, wherein executing the distributed map state comprises:
for each batch of a plurality of batches,
invoking, on a host computing device, a child execution of the second workflow with the batch as input to the second workflow, wherein the plurality of batches is based at least in part on a plurality of items and the batch parameter, wherein the plurality of items is based at least in part on the resource,
and wherein completion of the child execution of the second workflow on the host computing device generates output; and
receiving the output generated by the child execution of the second workflow on the host computing device; and
storing the output from each batch of the plurality of batches.
8 . The computer-implemented method of claim 7 , wherein the second workflow comprises a first state and a second state, wherein output from the first state as input to the second state is limited to a first data input size, and wherein a data size of a collection of item identifiers for the plurality of items exceeds the first data input size.
9 . The computer-implemented method of claim 7 , wherein
the output from each batch of the plurality of batches is stored in user data storage.
10 . The computer-implemented method of claim 7 , wherein the resource identifies a location in a second data store, and wherein the method further comprises:
loading a collection of item identifiers from the location in the second data store.
11 . The computer-implemented method of claim 7 , wherein at least one item of the plurality of items comprises an item identifier.
12 . The computer-implemented method of claim 7 ,
wherein child executions of the second workflow on devices in a quantity of host computing devices execute concurrently, and wherein the quantity of host computing devices is based at least in part on a count of the plurality of batches.
13 . The computer-implemented method of claim 12 , wherein the quantity of host computing devices matches the count of the plurality of batches.
14 . A system comprising:
a data store storing computer-executable instructions; and
one or more computer hardware processors in communication with the data store, wherein the one or more computer hardware processors are configured to execute the computer-executable instructions to at least:
receive a first workflow comprising a first distributed map state, the first distributed map state comprising a resource, a batch parameter, and a second workflow; and
execute the first distributed map state, wherein to execute the first distributed map state, the one or more computer hardware processors are configured to execute the computer-executable instructions to at least:
for each batch of a plurality of batches,
invoke, on a host computing device, a child execution of the second workflow with the batch as input to the second workflow, wherein the plurality of batches is based at least in part on a plurality of items and the batch parameter, wherein the plurality of items is based at least in part on the resource, and wherein completion of the child execution of the second workflow on the host computing device generates output; and
receive the output generated by the child execution of the second workflow on the host computing device; and
store the output from each batch of the plurality of batches.
15 . The system of claim 14 , wherein the resource identifies a second data store, and wherein the one or more computer hardware processors are configured to execute further computer-executable instructions to at least:
load a collection of item identifiers from the second data store.
16 . The system of claim 14 , wherein the resource identifies an Application Programming Interface (API), and wherein the one or more computer hardware processors are configured to execute further computer-executable instructions to at least:
call the API; and
receive, from the API, a collection of item identifiers.
17 . The system of claim 14 , wherein the batch parameter is a data size,
wherein a total data size for each group of a plurality of groups of items, from the plurality of items, satisfies a threshold value,
wherein the threshold value is based at least in part on the data size, and
wherein each group from the plurality of groups of items is a batch.
18 . The system of claim 14 ,
wherein child executions of the second workflow on devices in a quantity of host computing devices execute concurrently, and wherein the quantity is based at least in part on a count of the plurality of batches.
19 . The system of claim 18 , wherein the quantity of host computing devices corresponds to the count of the plurality of batches.
20 . The system of claim 18 , wherein
the count of the plurality of batches fails to satisfy a concurrency criterion and
the quantity of host computing devices executing concurrently is associated with a concurrency number.
21 . The system of claim 14 , wherein the first distributed map state comprises a second distributed map state, the second distributed map state comprising a third workflow, the one or more computer hardware processors are configured to execute further computer-executable instructions to at least:
for each second batch of a second plurality of batches,
invoke, on a second host computing device, a child execution of the third workflow with the second batch as input to the third workflow, wherein completion of the child execution of the third workflow on the second host computing device generates second output, and wherein the second plurality of batches is based at least in part on the output from each batch of the plurality of batches; and
receive the second output generated by the child execution of the third workflow on the second host computing device; and
store the second output from each second batch of the second plurality of batches.