Data object preparation for execution of multiple task routine instances in many task computing
An apparatus includes a processor to: output a request message to cause a first task to be performed; within a task container, in response to the request message and a data object not being divided, divide the data object into a set of data object blocks based on at least the sizes of the data object and the atomic unit of organization of data therein, as well as the storage resources allocated to task containers, and output a task completion message indicating that the first task has been performed, and including a set of data block identifiers indicating the location of the set of data object blocks within at least one federated area; and in response to the task completion message, output a set of request messages to cause a second task to be performed by executing multiple instances of a task routine within multiple task containers.
1. An apparatus comprising at least one processor and a storage to store instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
receive, at the at least one processor, and from a requesting device via a network, a request to perform a job flow comprising a set of tasks;
within a performance container, the at least one processor is caused to output a first task routine execution request message;
within a first task container, and in response to the first task routine execution request message, the at least one processor is caused to perform operations of a first task comprising:
access a first data object within at least one federated area to determine whether the first data object is already divided into a first set of data object blocks;
in response to a determination that the first data object is not already divided, perform operations comprising:
analyze the first data object to determine a size of the first data object;
analyze a data structure by which data values are organized within the first data object to identify an atomic unit of storage of data values within the data structure, and to determine a size of the atomic unit;
based on at least the size of the first data object, the size of the atomic unit, and storage resources allocated to task containers, determine a quantity of data object blocks into which to divide the first data object;
divide the first data object into the quantity of data object blocks to generate the first set of data object blocks; and
output a first task completion message comprising a first set of data block identifiers, wherein each data block identifier of the first set of data block identifiers indicates a location within the at least one federated area at which a different data object block of the first set of data object blocks is stored; and
in response to a determination that the first data object is already divided, perform operations comprising:
retrieve the first set of data block identifiers from the at least one federated area; and
output the first task completion message comprising the first set of data block identifiers; and
within the performance container, and in response to the first task completion message, the at least one processor is caused to output a first set of task routine execution request messages to cause a second task to be performed by executing multiple instances of a task routine within multiple task containers at least partially in parallel, wherein:
each task routine execution request message of the first set of task routine execution request messages includes a different data block identifier of the first set of data block identifiers to cause the at least one processor to execute each instance of the task routine using a different data object block of the first set of data object blocks as an input.
2. The apparatus of claim 1 , wherein, prior to receiving the request to perform the job flow, the at least one processor is caused to perform operations comprising:
receive, at the at least one processor, and from another requesting device via the network, an earlier request to store the first data object within the at least one federated area;
compare the size of the first data object to a threshold size associated with a limitation imposed on data objects stored within the at least one federated area; and
in response to a determination that the size of the first data object is larger than the threshold size, perform operations comprising:
analyze the first data object to determine whether the first data object is in a distributable form in which data values within the first data object are organized into a single homogeneous data structure;
in response to a determination that the first data object is not in distributable form, reorganize the data values within the first data object into a single homogenous data structure to convert the first data object into distributable form;
with the first data object in distributable form, divide the first data object into the first set of data object blocks; and
store the first set of data object blocks within the at least one federated area at locations indicated by the first set of data block identifiers.
3. The apparatus of claim 1 , wherein, at a time prior to receiving the request to perform the job flow, the first data object was generated in distributed form as the first set of data object blocks as an output of a performance of another task of another job flow.
4. The apparatus of claim 1 , wherein dividing the first data object into the quantity of data objects comprises the at least one processor performing operations within the first task container to define the first set of data object blocks as overlying the first data object as already stored within the at least one federated area, the operations comprising:
determine a quantity of atomic units of storage of data values within the data structure to allocate to each data object block of the first set of data object blocks;
based on at least the quantity of atomic units per data object block, determine each location within the data structure at which to define a division between two adjacent atomic units that defines a boundary between two adjacent data object blocks of the first set of data object blocks;
identify where each boundary between two adjacent data object blocks is located within the first data object as already stored within the at least one federated area as a single undivided data object; and
generate the first set of data block identifiers to indicate the location within the at least one federated area at which the first data object begins, and to indicate each location within the at least one federated area of a boundary between two adjacent data object blocks of the first set of data object blocks.
5. The apparatus of claim 1 , wherein dividing the first data object into the quantity of data objects comprises the at least one processor performing operations within the first task container to store the first data object within the at least one federated area separately from the first data object as already stored within the at least one federated area, the operations comprising:
determine a quantity of atomic units of storage of data values within the data structure to allocate to each data object block of the first set of data object blocks;
based on at least the quantity of atomic units per data object block, determine each location within the data structure at which to define a division between two adjacent atomic units that defines a boundary between two adjacent data object blocks of the first set of data object blocks; and
store the first set of data object blocks within the at least one federated area at locations indicated by the first set of data block identifiers, wherein the locations indicated by the first set of data block identifiers do not overlie the location at which the first data object is already stored as a single undivided object.
6. The apparatus of claim 1 , wherein:
a third task of the set of tasks of the job flow combines a second set of data object blocks of a second data object in distributed form to generate an undivided single object form of the second data object as an output;
within the performance container, the at least one processor is caused to output a second task routine execution request message to cause the third task to be performed, wherein:
the second task routine execution request message includes a second set of data block identifiers that indicate locations at which the second set of data object blocks are stored within the at least one federated area; and
within a second task container, the at least one processor is caused to perform operations of the third task comprising:
use the second set of data block identifiers included in the second task routine execution request message to retrieve the second set of data object blocks;
combine the second set of data object blocks to generate the second data object as a single undivided data object; and
store the second data object in the at least one federated area.
7. The apparatus of claim 1 , wherein:
the second task uses the first data object as an input to generate a second data object as an output; and
within each task container of the multiple task containers, and in response to one of the task routine execution request messages of the first set of task routine execution request messages, the at least one processor is caused to perform operations of the second task comprising:
use the data block identifier included in the one of the task routine execution request messages to retrieve a corresponding data object block of the first set of data object blocks;
execute a corresponding instance of the task routine of the multiple instances of the task routine to use the retrieved data object block of the first set of data object blocks as an input to generate a corresponding data object block of a second set of data object blocks of the second data object as an output;
store the output data object block of the second set of data object blocks within the at least one federated area at a location indicated by a data block identifier of a second set of data block identifiers; and
output a task completion message of a first set of task completion messages comprising the data block identifier of the second set of data block identifiers.
8. The apparatus of claim 7 , wherein:
a third task of the set of tasks of the job flow uses the second data object as an input;
within the performance container, and in response to a single task completion message of the first set of task completion messages output from a single task container of the multiple task containers in which the second task is performed, the at least one processor is caused to perform operations comprising:
provide an indication to the single task container to await output of another task routine execution request message directed to the single task container to cause another task to be performed within the single task container; and
output, to the single task container, a task routine execution request message of a second set of task routine execution request messages to cause the third task to be performed within the single task container using the data object block of the second set of data object blocks, wherein:
the single task routine execution request message includes the data block identifier that is included in the single task completion message; and
within the single task container, and in response to the single task routine execution message, the at least one processor is caused to perform operations of the third task comprising:
execute an instance of a third task routine of multiple instances of the third task routine to use the data object block of the second set of data object blocks that was generated within the single task container as an input; and
output a task completion message of a second set of task completion messages to indicate completion of the third task within the single task container.
9. The apparatus of claim 1 , wherein:
the job flow is defined in a job flow definition that specifies a set of tasks to be performed by executing a corresponding set of task routines, and that specifies data dependencies among the set of tasks;
the set of tasks comprises the first task and the second task;
the job flow definition, the set of tasks and the first data object are stored within the at least one federated area; and
within the performance container, the at least one processor is caused to perform operations comprising:
derive an order of performance of the set of tasks based on the data dependencies among the set of tasks; and
output the first task routine execution request message to cause the performance of the first task, and output the first set of task routine execution request messages to cause the performance of the second task based on the order of performance of the set of tasks.
10. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause at least one processor to perform operations comprising:
receive, at the at least one processor, and from a requesting device via a network, a request to perform a job flow comprising a set of tasks;
within a performance container, the at least one processor is caused to output a first task routine execution request message;
within a first task container, and in response to the first task routine execution request message, the at least one processor is caused to perform operations of a first task comprising:
access a first data object within at least one federated area to determine whether the first data object is already divided into a first set of data object blocks;
in response to a determination that the first data object is not already divided, perform operations comprising:
analyze the first data object to determine a size of the first data object;
analyze a data structure by which data values are organized within the first data object to identify an atomic unit of storage of data values within the data structure, and to determine a size of the atomic unit;
based on at least the size of the first data object, the size of the atomic unit, and storage resources allocated to task containers, determine a quantity of data object blocks into which to divide the first data object;
divide the first data object into the quantity of data object blocks to generate the first set of data object blocks; and
output a first task completion message comprising a first set of data block identifiers, wherein each data block identifier of the first set of data block identifiers indicates a location within the at least one federated area at which a different data object block of the first set of data object blocks is stored; and
in response to a determination that the first data object is already divided, perform operations comprising:
retrieve the first set of data block identifiers from the at least one federated area; and
output the first task completion message comprising the first set of data block identifiers; and
within the performance container, and in response to the first task completion message, the at least one processor is caused to output a first set of task routine execution request messages to cause a second task to be performed by executing multiple instances of a task routine within multiple task containers at least partially in parallel, wherein:
each task routine execution request message of the first set of task routine execution request messages includes a different data block identifier of the first set of data block identifiers to cause the at least one processor to execute each instance of the task routine using a different data object block of the first set of data object blocks as an input.
11. The computer-program product of claim 10 , wherein, prior to receiving the request to perform the job flow, the at least one processor is caused to perform operations comprising:
receive, at the at least one processor, and from another requesting device via the network, an earlier request to store the first data object within the at least one federated area;
compare the size of the first data object to a threshold size associated with a limitation imposed on data objects stored within the at least one federated area; and
in response to a determination that the size of the first data object is larger than the threshold size, perform operations comprising:
analyze the first data object to determine whether the first data object is in a distributable form in which data values within the first data object are organized into a single homogeneous data structure;
in response to a determination that the first data object is not in distributable form, reorganize the data values within the first data object into a single homogenous data structure to convert the first data object into distributable form;
with the first data object in distributable form, divide the first data object into the first set of data object blocks; and
store the first set of data object blocks within the at least one federated area at locations indicated by the first set of data block identifiers.
12. The computer-program product of claim 10 , wherein, at a time prior to receiving the request to perform the job flow, the first data object was generated in distributed form as the first set of data object blocks as an output of a performance of another task of another job flow.
13. The computer-program product of claim 10 , wherein dividing the first data object into the quantity of data objects comprises the at least one processor performing operations within the first task container to define the first set of data object blocks as overlying the first data object as already stored within the at least one federated area, the operations comprising:
determine a quantity of atomic units of storage of data values within the data structure to allocate to each data object block of the first set of data object blocks;
based on at least the quantity of atomic units per data object block, determine each location within the data structure at which to define a division between two adjacent atomic units that defines a boundary between two adjacent data object blocks of the first set of data object blocks;
identify where each boundary between two adjacent data object blocks is located within the first data object as already stored within the at least one federated area as a single undivided data object; and
generate the first set of data block identifiers to indicate the location within the at least one federated area at which the first data object begins, and to indicate each location within the at least one federated area of a boundary between two adjacent data object blocks of the first set of data object blocks.
14. The computer-program product of claim 10 , wherein dividing the first data object into the quantity of data objects comprises the at least one processor performing operations within the first task container to store the first data object within the at least one federated area separately from the first data object as already stored within the at least one federated area, the operations comprising:
determine a quantity of atomic units of storage of data values within the data structure to allocate to each data object block of the first set of data object blocks;
based on at least the quantity of atomic units per data object block, determine each location within the data structure at which to define a division between two adjacent atomic units that defines a boundary between two adjacent data object blocks of the first set of data object blocks; and
store the first set of data object blocks within the at least one federated area at locations indicated by the first set of data block identifiers, wherein the locations indicated by the first set of data block identifiers do not overlie the location at which the first data object is already stored as a single undivided object.
15. The computer-program product of claim 10 , wherein:
a third task of the set of tasks of the job flow combines a second set of data object blocks of a second data object in distributed form to generate an undivided single object form of the second data object as an output;
within the performance container, the at least one processor is caused to output a second task routine execution request message to cause the third task to be performed, wherein:
the second task routine execution request message includes a second set of data block identifiers that indicate locations at which the second set of data object blocks are stored within the at least one federated area; and
within a second task container, the at least one processor is caused to perform operations of the third task comprising:
use the second set of data block identifiers included in the second task routine execution request message to retrieve the second set of data object blocks;
combine the second set of data object blocks to generate the second data object as a single undivided data object; and
store the second data object in the at least one federated area.
16. The computer-program product of claim 10 , wherein:
the second task uses the first data object as an input to generate a second data object as an output; and
within each task container of the multiple task containers, and in response to one of the task routine execution request messages of the first set of task routine execution request messages, the at least one processor is caused to perform operations of the second task comprising:
use the data block identifier included in the one of the task routine execution request messages to retrieve a corresponding data object block of the first set of data object blocks;
execute a corresponding instance of the task routine of the multiple instances of the task routine to use the retrieved data object block of the first set of data object blocks as an input to generate a corresponding data object block of a second set of data object blocks of the second data object as an output;
store the output data object block of the second set of data object blocks within the at least one federated area at a location indicated by a data block identifier of a second set of data block identifiers; and
output a task completion message of a first set of task completion messages comprising the data block identifier of the second set of data block identifiers.
17. The computer-program product of claim 16 , wherein:
a third task of the set of tasks of the job flow uses the second data object as an input;
within the performance container, and in response to a single task completion message of the first set of task completion messages output from a single task container of the multiple task containers in which the second task is performed, the at least one processor is caused to perform operations comprising:
provide an indication to the single task container to await output of another task routine execution request message directed to the single task container to cause another task to be performed within the single task container; and
output, to the single task container, a task routine execution request message of a second set of task routine execution request messages to cause the third task to be performed within the single task container using the data object block of the second set of data object blocks, wherein:
the single task routine execution request message includes the data block identifier that is included in the single task completion message; and
within the single task container, and in response to the single task routine execution message, the at least one processor is caused to perform operations of the third task comprising:
execute an instance of a third task routine of multiple instances of the third task routine to use the data object block of the second set of data object blocks that was generated within the single task container as an input; and
output a task completion message of a second set of task completion messages to indicate completion of the third task within the single task container.
18. The computer-program product of claim 10 , wherein:
the job flow is defined in a job flow definition that specifies a set of tasks to be performed by executing a corresponding set of task routines, and that specifies data dependencies among the set of tasks;
the set of tasks comprises the first task and the second task;
the job flow definition, the set of tasks and the first data object are stored within the at least one federated area; and
within the performance container, the at least one processor is caused to perform operations comprising:
derive an order of performance of the set of tasks based on the data dependencies among the set of tasks; and
output the first task routine execution request message to cause the performance of the first task, and output the first set of task routine execution request messages to cause the performance of the second task based on the order of performance of the set of tasks.
19. A computer-implemented method comprising:
receiving, by at the at least one processor, and from a requesting device via a network, a request to perform a job flow comprising a set of tasks;
within a performance container, outputting a first task routine execution request message;
within a first task container, and in response to the first task routine execution request message, performing operations of a first task comprising:
accessing a first data object within at least one federated area to determine, by the at least one processor, whether the first data object is already divided into a first set of data object blocks;
in response to a determination that the first data object is not already divided, performing operations comprising:
analyzing, by the at least one processor, the first data object to determine a size of the first data object;
analyzing, by the at least one processor, a data structure by which data values are organized within the first data object to identify an atomic unit of storage of data values within the data structure, and to determine a size of the atomic unit;
based on at least the size of the first data object, the size of the atomic unit, and storage resources allocated to task containers, determining, by the at least one processor, a quantity of data object blocks into which to divide the first data object;
dividing the first data object into the quantity of data object blocks to generate the first set of data object blocks; and
outputting a first task completion message comprising a first set of data block identifiers, wherein each data block identifier of the first set of data block identifiers indicates a location within the at least one federated area at which a different data object block of the first set of data object blocks is stored; or
in response to a determination that the first data object is already divided, performing operations comprising:
retrieving the first set of data block identifiers from the at least one federated area; and
outputting the first task completion message comprising the first set of data block identifiers; and
within the performance container, and in response to the first task completion message, outputting a first set of task routine execution request messages to cause a second task to be performed by executing multiple instances of a task routine within multiple task containers at least partially in parallel, wherein:
each task routine execution request message of the first set of task routine execution request messages includes a different data block identifier of the first set of data block identifiers to cause the at least one processor to execute each instance of the task routine using a different data object block of the first set of data object blocks as an input.
20. The computer-implemented method of claim 19 , comprising, prior to receiving the request to perform the job flow, performing operations comprising:
receiving, at the at least one processor, and from another requesting device via the network, an earlier request to store the first data object within the at least one federated area;
comparing, by the at least one processor, the size of the first data object to a threshold size associated with a limitation imposed on data objects stored within the at least one federated area; and
in response to a determination that the size of the first data object is larger than the threshold size, performing operations comprising:
analyzing, by the at least one processor, the first data object to determine whether the first data object is in a distributable form in which data values within the first data object are organized into a single homogeneous data structure;
in response to a determination that the first data object is not in distributable form, reorganizing, by the at least one processor, the data values within the first data object into a single homogenous data structure to convert the first data object into distributable form;
with the first data object in distributable form, dividing, by the at least one processor, the first data object into the first set of data object blocks; and
storing the first set of data object blocks within the at least one federated area at locations indicated by the first set of data block identifiers.
21. The computer-implemented method of claim 19 , comprising, at a time prior to receiving the request to perform the job flow, generating the first data object in distributed form as the first set of data object blocks as an output of a performance of another task of another job flow.
22. The computer-implemented method of claim 19 , wherein dividing the first data object into the quantity of data objects comprises performing operations within the first task container to define the first set of data object blocks as overlying the first data object as already stored within the at least one federated area, the operations comprising:
determining, by the at least one processor, a quantity of atomic units of storage of data values within the data structure to allocate to each data object block of the first set of data object blocks;
based on at least the quantity of atomic units per data object block, determining, by the at least one processor, each location within the data structure at which to define a division between two adjacent atomic units that defines a boundary between two adjacent data object blocks of the first set of data object blocks;
identifying, by the at least one processor, where each boundary between two adjacent data object blocks is located within the first data object as already stored within the at least one federated area as a single undivided data object; and
generating, by the at least one processor, the first set of data block identifiers to indicate the location within the at least one federated area at which the first data object begins, and to indicate each location within the at least one federated area of a boundary between two adjacent data object blocks of the first set of data object blocks.
23. The computer-implemented method of claim 19 , wherein dividing the first data object into the quantity of data objects comprises performing operations within the first task container to store the first data object within the at least one federated area separately from the first data object as already stored within the at least one federated area, the operations comprising:
determining, by the at least one processor, a quantity of atomic units of storage of data values within the data structure to allocate to each data object block of the first set of data object blocks;
based on at least the quantity of atomic units per data object block, determining, by the at least one processor, each location within the data structure at which to define a division between two adjacent atomic units that defines a boundary between two adjacent data object blocks of the first set of data object blocks; and
storing the first set of data object blocks within the at least one federated area at locations indicated by the first set of data block identifiers, wherein the locations indicated by the first set of data block identifiers do not overlie the location at which the first data object is already stored as a single undivided object.
24. The computer-implemented method of claim 19 , wherein:
a third task of the set of tasks of the job flow combines a second set of data object blocks of a second data object in distributed form to generate an undivided single object form of the second data object as an output; and
the method further comprises:
within the performance container, outputting a second task routine execution request message to cause the third task to be performed, wherein:
the second task routine execution request message includes a second set of data block identifiers that indicate locations at which the second set of data object blocks are stored within the at least one federated area; and
within a second task container, performing operations of the third task comprising:
using the second set of data block identifiers included in the second task routine execution request message to retrieve the second set of data object blocks;
combining, by the at least one processor, the second set of data object blocks to generate the second data object as a single undivided data object; and
storing the second data object in the at least one federated area.
25. The computer-implemented method of claim 19 , wherein:
the second task uses the first data object as an input to generate a second data object as an output; and
the method further comprises, within each task container of the multiple task containers, and in response to one of the task routine execution request messages of the first set of task routine execution request messages, performing operations of the second task comprising:
using the data block identifier included in the one of the task routine execution request messages to retrieve a corresponding data object block of the first set of data object blocks;
executing, by the at least one processor, a corresponding instance of the task routine of the multiple instances of the task routine to use the retrieved data object block of the first set of data object blocks as an input to generate a corresponding data object block of a second set of data object blocks of the second data object as an output;
storing the output data object block of the second set of data object blocks within the at least one federated area at a location indicated by a data block identifier of a second set of data block identifiers; and
outputting a task completion message of a first set of task completion messages comprising the data block identifier of the second set of data block identifiers.
26. The computer-implemented method of claim 25 , wherein:
a third task of the set of tasks of the job flow uses the second data object as an input; and
the method further comprises:
within the performance container, and in response to a single task completion message of the first set of task completion messages output from a single task container of the multiple task containers in which the second task is performed, performing operations comprising:
providing an indication to the single task container to await output of another task routine execution request message directed to the single task container to cause another task to be performed within the single task container; and
outputting, to the single task container, a task routine execution request message of a second set of task routine execution request messages to cause the third task to be performed within the single task container using the data object block of the second set of data object blocks, wherein:
the single task routine execution request message includes the data block identifier that is included in the single task completion message; and
within the single task container, and in response to the single task routine execution message, performing operations of the third task comprising:
executing, by the at least one processor, an instance of a third task routine of multiple instances of the third task routine to use the data object block of the second set of data object blocks that was generated within the single task container as an input; and
outputting a task completion message of a second set of task completion messages to indicate completion of the third task within the single task container.
27. The computer-implemented method of claim 19 , wherein:
the job flow is defined in a job flow definition that specifies a set of tasks to be performed by executing a corresponding set of task routines, and that specifies data dependencies among the set of tasks;
the set of tasks comprises the first task and the second task;
the job flow definition, the set of tasks and the first data object are stored within the at least one federated area; and
the method comprises, within the performance container, performing operations comprising:
deriving, by the at least one processor, an order of performance of the set of tasks based on the data dependencies among the set of tasks; and
outputting the first task routine execution request message to cause the performance of the first task, and output the first set of task routine execution request messages to cause the performance of the second task based on the order of performance of the set of tasks.