Many task computing with message passing interface
An apparatus includes a processor to: receive a request to perform a set of tasks; analyze input data objects used as inputs and output data objects generated as outputs by the set of tasks to identify at least one dependency between tasks and at least one pair of tasks performable in parallel; determine an order of performance of the set of tasks; retrieve executable instructions of a corresponding set of task routines that define input interfaces and output interfaces for the input data objects and output data objects, respectively; generate a job flow definition defining the order of performance, each input interface and each output interface; use the job flow definition to perform the set of tasks; and transmit one of the output data objects to the requesting device as a result report serving as the output of the set of tasks.
1. An apparatus comprising a processor and a storage to store instructions that, when executed by the processor, cause the processor to perform operations comprising:
receive, at the processor, and from a remote device via a network, a request to perform a set of tasks, wherein, for each task of the set of tasks:
an input data object is specified in the request to be used by the task as an input; and
an output data object is specified in the request to be generated by the task as an output;
analyze the specification of an input data object and the specification of an output data object for each task of the set of tasks to identify at least one dependency among at least two tasks in which the output data object specified for one task becomes the input data object specified for another task;
analyze the specification of an input data object and the specification of an output data object for each task of the set of tasks to identify at least one pair of tasks able to be performed in parallel due to a lack of dependency therebetween;
determine an order of performance of the set of tasks based on the at least one dependency and based on the at least one pair of tasks that are able to be performed in parallel in the order due to the lack of dependency therebetween;
retrieve, for each task of the set of tasks, and from at least one federated area, a most recent version of a corresponding task routine that comprises executable instructions to perform the task;
retrieve, for each task of the set of tasks, and from the executable instructions of the corresponding task routine, executable instructions that implement an input interface by which the input data object specified for the task is received, and executable instructions that implement an output interface by which the output data object specified for the task is generated;
generate, within the at least one federated area, a job flow definition comprising:
a specification of the order of performance of the set of tasks, including the ability to perform the at least one pair of tasks in parallel;
for each task of the set of tasks, a specification of the input interface by which the task receives the input data object specified for the task; and
for each task of the set of tasks, a specification of the output interface by which the task generates the output data object specified for the task;
use the specification of the order of performance of the set of tasks, the specification of the input interface for each task of the set of tasks, and the specification of the output interface for each task of the set of tasks in the job flow definition, along with each input data object and each output data object specified in the request, to perform the set of tasks specified in the request; and
transmit an output data object generated by a task of the set of tasks as a result report output by the set of tasks to the remote device.
2. The apparatus of claim 1 , wherein the processor is caused to perform operations comprising:
analyze the specification of an input data object and the specification of an output data object for each task of the set of tasks to identify at least one error in dependency in which a input data object is required to be available as an input to a task before being able to be generated as an output data object; and
in response to the identification of the at least one error in dependency, refrain from performing the set of tasks, and transmit an indication of the at least one error in dependency to the remote device in lieu of the transmission of the result report.
3. The apparatus of claim 1 , wherein the processor is caused to perform operations comprising:
analyze the specification of an input data object and the specification of an output data object for each task of the set of tasks to identify at least one error in dependency in which an output interface by which one task is to generate an output data object that is to be used as an input data object to another task through an input interface that does not match the output interface; and
in response to the identification of the at least one error in dependency, refrain from performing the set of tasks, and transmit an indication of the at least one error in dependency to the remote device in lieu of the transmission of the result report.
4. The apparatus of claim 1 , wherein, for at least one task of the set of tasks, the processor is caused to perform operations comprising:
determine whether the input data object specified for at least one task is stored as an undivided object or as a set of data object blocks distributed among a set of devices;
in response to a determination that the input data object is stored as a set of data object blocks, determine whether the input data object is stored in a distributable form that enables multiple instances of the at least one task to be performed independently and at least partially in parallel with each instance using a data object block of the set of data object blocks as an input to the instance; and
in response to the determination that the input data object is stored as a set of data object blocks and a determination that the input data object is stored in distributable form, perform operations comprising:
generate a container that contains at least one task routine of the retrieved task routines to enable a processor incorporated into each device of the set of devices to independently perform an instance of the multiple instances of the at least one task using one of the data object blocks of the data object stored locally within the device as an input to the instance, wherein the performance of an instance of the at least one task within each device generates a corresponding data object block of a set of data object blocks of an output data object; and
provide a copy of the container to each device of the set of devices to enable the processors incorporated into least two devices of the set of devices to perform instances of the at least one task at least partially in parallel.
5. The apparatus of claim 4 , wherein:
the entirety of the output data object is to become an input data object used as an input to another task routine to be executed within another device; and
the processor is caused to perform operations comprising:
retrieve, from each device of the set of devices, at least one data object block of the set of data object blocks of the output data object;
assemble the output data object from the set of data object blocks of the output object; and
provide the output data object to the other device.
6. The apparatus of claim 5 , wherein:
the output data object is specified in the request as a data object that is to be preserved; and
the processor is caused, in response to the specification of the output data object as a data object that is to be preserved, to perform operations comprising:
compare a size of the output data object to a predetermined threshold size; and
store, based on the comparison, the output data object within as an undivided object within a federated area within a single device, or as a set of data object blocks within a federated area that spans multiple devices.
7. The apparatus of claim 4 , wherein:
each data object block of the output data object is to be used as an input to a corresponding instance of multiple instances of another task routine of the set of task routines; and
the processor is caused to include the task routine of the retrieved task routines that corresponds to the other task in the container to enable a processor incorporated into each device of the set of devices to independently perform an instance of the other task using the one of the data object blocks of the output data object stored locally within the device as an input to the instance.
8. The apparatus of claim 4 , wherein:
the output data object comprises the result report; and
the processor is caused to perform operations comprising:
retrieve, from each device of the set of devices, at least one data object block of the set of data object blocks of the output data object;
assemble the output data object from the set of data object blocks of the output data object;
store the output data object in the at least one federated area as the result report; and
transmit the output data object to the remote device as the result report.
9. The apparatus of claim 1 , wherein:
a data object is received as embedded within the request; and
the processor is caused to perform operations comprising:
compare a size of the received data object to a predetermined threshold size to determine whether the size of the received data object is larger than the predetermined threshold size; and
in response to a determination that the size of the received data object is larger than the predetermined threshold size, perform operations comprising:
analyze the flow input data set to determine whether the received data object is of a distributable form in which there is no distinct metadata structure therein, and in which data items of the received data object are organized into a single homogeneous data structure wherein the data items remain accessible from each data object block of a set of data object blocks into which the received data object may be divided independently of the other data object blocks after the division of the received data object;
in response to a determination that the received data object is not of the distributable form, convert the received data object block into the distributable form; and
store the received data object as the set of data object blocks within a federated area that spans multiple devices.
10. The apparatus of claim 9 , wherein, the processor is caused, in response to a determination that the received data object is smaller than the predetermined threshold size, store the received data object as an undivided object within a federated area within a single device.
11. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause a processor to perform operations comprising:
receive, at the processor, and from a remote device via a network, a request to perform a set of tasks, wherein, for each task of the set of tasks:
an input data object is specified in the request to be used by the task as an input; and
an output data object is specified in the request to be generated by the task as an output;
analyze the specification of an input data object and the specification of an output data object for each task of the set of tasks to identify at least one dependency among at least two tasks in which the output data object specified for one task becomes the input data object specified for another task;
analyze the specification of an input data object and the specification of an output data object for each task of the set of tasks to identify at least one pair of tasks able to be performed in parallel due to a lack of dependency therebetween;
determine an order of performance of the set of tasks based on the at least one dependency and based on the at least one pair of tasks that are able to be performed in parallel in the order due to the lack of dependency therebetween;
retrieve, for each task of the set of tasks, and from at least one federated area, a most recent version of a corresponding task routine that comprises executable instructions to perform the task;
retrieve, for each task of the set of tasks, and from the executable instructions of the corresponding task routine, executable instructions that implement an input interface by which the input data object specified for the task is received, and executable instructions that implement an output interface by which the output data object specified for the task is generated;
generate, within the at least one federated area, a job flow definition comprising:
a specification of the order of performance of the set of tasks, including the ability to perform the at least one pair of tasks in parallel;
for each task of the set of tasks, a specification of the input interface by which the task receives the input data object specified for the task; and
for each task of the set of tasks, a specification of the output interface by which the task generates the output data object specified for the task;
use the specification of the order of performance of the set of tasks, the specification of the input interface for each task of the set of tasks, and the specification of the output interface for each task of the set of tasks in the job flow definition, along with each input data object and each output data object specified in the request, to perform the set of tasks specified in the request; and
transmit an output data object generated by a task of the set of tasks as a result report output by the set of tasks to the remote device.
12. The computer-program product of claim 11 , wherein the processor is caused to perform operations comprising:
analyze the specification of an input data object and the specification of an output data object for each task of the set of tasks to identify at least one error in dependency in which a input data object is required to be available as an input to a task before being able to be generated as an output data object; and
in response to the identification of the at least one error in dependency, refrain from performing the set of tasks, and transmit an indication of the at least one error in dependency to the remote device in lieu of the transmission of the result report.
13. The computer-program product of claim 11 , wherein the processor is caused to perform operations comprising:
analyze the specification of an input data object and the specification of an output data object for each task of the set of tasks to identify at least one error in dependency in which an output interface by which one task is to generate an output data object that is to be used as an input data object to another task through an input interface that does not match the output interface; and
in response to the identification of the at least one error in dependency, refrain from performing the set of tasks, and transmit an indication of the at least one error in dependency to the remote device in lieu of the transmission of the result report.
14. The computer-program product of claim 11 , wherein, for at least one task of the set of tasks, the processor is caused to perform operations comprising:
determine whether the input data object specified for at least one task is stored as an undivided object or as a set of data object blocks distributed among a set of devices;
in response to a determination that the input data object is stored as a set of data object blocks, determine whether the input data object is stored in a distributable form that enables multiple instances of the at least one task to be performed independently and at least partially in parallel with each instance using a data object block of the set of data object blocks as an input to the instance; and
in response to the determination that the input data object is stored as a set of data object blocks and a determination that the input data object is stored in distributable form, perform operations comprising:
generate a container that contains at least one task routine of the retrieved task routines to enable a processor incorporated into each device of the set of devices to independently perform an instance of the multiple instances of the at least one task using one of the data object blocks of the data object stored locally within the device as an input to the instance, wherein the performance of an instance of the at least one task within each device generates a corresponding data object block of a set of data object blocks of an output data object; and
provide a copy of the container to each device of the set of devices to enable the processors incorporated into least two devices of the set of devices to perform instances of the at least one task at least partially in parallel.
15. The computer-program product of claim 14 , wherein:
the entirety of the output data object is to become an input data object used as an input to another task routine to be executed within another device; and
the processor is caused to perform operations comprising:
retrieve, from each device of the set of devices, at least one data object block of the set of data object blocks of the output data object;
assemble the output data object from the set of data object blocks of the output object; and
provide the output data object to the other device.
16. The computer-program product of claim 15 , wherein:
the output data object is specified in the request as a data object that is to be preserved; and
the processor is caused, in response to the specification of the output data object as a data object that is to be preserved, to perform operations comprising:
compare a size of the output data object to a predetermined threshold size; and
store, based on the comparison, the output data object within as an undivided object within a federated area within a single device, or as a set of data object blocks within a federated area that spans multiple devices.
17. The computer-program product of claim 14 , wherein:
each data object block of the output data object is to be used as an input to a corresponding instance of multiple instances of another task routine of the set of task routines; and
the processor is caused to include the task routine of the retrieved task routines that corresponds to the other task in the container to enable a processor incorporated into each device of the set of devices to independently perform an instance of the other task using the one of the data object blocks of the output data object stored locally within the device as an input to the instance.
18. The computer-program product of claim 14 , wherein:
the output data object comprises the result report; and
the processor is caused to perform operations comprising:
retrieve, from each device of the set of devices, at least one data object block of the set of data object blocks of the output data object;
assemble the output data object from the set of data object blocks of the output data object;
store the output data object in the at least one federated area as the result report; and
transmit the output data object to the remote device as the result report.
19. The computer-program product of claim 11 , wherein:
a data object is received as embedded within the request; and
the processor is caused to perform operations comprising:
compare a size of the received data object to a predetermined threshold size to determine whether the size of the received data object is larger than the predetermined threshold size; and
in response to a determination that the size of the received data object is larger than the predetermined threshold size, perform operations comprising:
analyze the flow input data set to determine whether the received data object is of a distributable form in which there is no distinct metadata structure therein, and in which data items of the received data object are organized into a single homogeneous data structure wherein the data items remain accessible from each data object block of a set of data object blocks into which the received data object may be divided independently of the other data object blocks after the division of the received data object;
in response to a determination that the received data object is not of the distributable form, convert the received data object block into the distributable form; and
store the received data object as the set of data object blocks within a federated area that spans multiple devices.
20. The computer-program product of claim 19 , wherein, the processor is caused, in response to a determination that the received data object is smaller than the predetermined threshold size, store the received data object as an undivided object within a federated area within a single device.
21. A computer-implemented method comprising:
receiving, by a processor, and from a remote device via a network, a request to perform a set of tasks, wherein, for each task of the set of tasks:
an input data object is specified in the request to be used by the task as an input; and
an output data object is specified in the request to be generated by the task as an output;
analyzing, by the processor, the specification of an input data object and the specification of an output data object for each task of the set of tasks to identify at least one dependency among at least two tasks in which the output data object specified for one task becomes the input data object specified for another task;
analyzing, by the processor, the specification of an input data object and the specification of an output data object for each task of the set of tasks to identify at least one pair of tasks able to be performed in parallel due to a lack of dependency therebetween;
determining, by the processor, an order of performance of the set of tasks based on the at least one dependency and based on the at least one pair of tasks that are able to be performed in parallel in the order due to the lack of dependency therebetween;
retrieving, for each task of the set of tasks, and from at least one federated area, a most recent version of a corresponding task routine that comprises executable instructions to perform the task;
retrieving, for each task of the set of tasks, and from the executable instructions of the corresponding task routine, executable instructions that implement an input interface by which the input data object specified for the task is received, and executable instructions that implement an output interface by which the output data object specified for the task is generated;
generating, by the processor, within the at least one federated area, a job flow definition comprising:
a specification of the order of performance of the set of tasks, including the ability to perform the at least one pair of tasks in parallel;
for each task of the set of tasks, a specification of the input interface by which the task receives the input data object specified for the task; and
for each task of the set of tasks, a specification of the output interface by which the task generates the output data object specified for the task;
using, by the processor, the specification of the order of performance of the set of tasks, the specification of the input interface for each task of the set of tasks, and the specification of the output interface for each task of the set of tasks in the job flow definition, along with each input data object and each output data object specified in the request, to perform the set of tasks specified in the request; and
transmitting, from the processor, an output data object generated by a task of the set of tasks as a result report output by the set of tasks to the remote device.
22. The computer-implemented method of claim 21 , comprising:
analyzing, by the processor, the specification of an input data object and the specification of an output data object for each task of the set of tasks to identify at least one error in dependency in which a input data object is required to be available as an input to a task before being able to be generated as an output data object; and
in response to the identification of the at least one error in dependency, refraining, by the processor, from performing the set of tasks, and transmitting, from the processor, an indication of the at least one error in dependency to the remote device in lieu of the transmission of the result report.
23. The computer-implemented method of claim 21 , comprising:
analyzing, by the processor, the specification of an input data object and the specification of an output data object for each task of the set of tasks to identify at least one error in dependency in which an output interface by which one task is to generate an output data object that is to be used as an input data object to another task through an input interface that does not match the output interface; and
in response to the identification of the at least one error in dependency, refraining, by the processor, from performing the set of tasks, and transmitting, from the processor, an indication of the at least one error in dependency to the remote device in lieu of the transmission of the result report.
24. The computer-implemented method of claim 21 , comprising, for at least one task of the set of tasks:
determining, by the processor, whether the input data object specified for at least one task is stored as an undivided object or as a set of data object blocks distributed among a set of devices;
in response to a determination that the input data object is stored as a set of data object blocks, determining, by the processor, whether the input data object is stored in a distributable form that enables multiple instances of the at least one task to be performed independently and at least partially in parallel with each instance using a data object block of the set of data object blocks as an input to the instance; and
in response to the determination that the input data object is stored as a set of data object blocks and a determination that the input data object is stored in distributable form, performing operations comprising:
generating, by the processor, a container that contains at least one task routine of the retrieved task routines to enable a processor incorporated into each device of the set of devices to independently perform an instance of the multiple instances of the at least one task using one of the data object blocks of the data object stored locally within the device as an input to the instance, wherein the performance of an instance of the at least one task within each device generates a corresponding data object block of a set of data object blocks of an output data object; and
providing a copy of the container to each device of the set of devices to enable the processors incorporated into least two devices of the set of devices to perform instances of the at least one task at least partially in parallel.
25. The computer-implemented method of claim 24 , wherein:
the entirety of the output data object is to become an input data object used as an input to another task routine to be executed within another device; and
the method comprises:
retrieving, from each device of the set of devices, at least one data object block of the set of data object blocks of the output data object;
assembling, by the processor, the output data object from the set of data object blocks of the output object; and
providing the output data object to the other device.
26. The computer-implemented method of claim 25 , wherein:
the output data object is specified in the request as a data object that is to be preserved; and
the method comprises, in response to the specification of the output data object as a data object that is to be preserved, performing operations comprising:
comparing, by the processor, a size of the output data object to a predetermined threshold size; and
storing, based on the comparison, the output data object within as an undivided object within a federated area within a single device, or as a set of data object blocks within a federated area that spans multiple devices.
27. The computer-implemented method of claim 24 , wherein:
each data object block of the output data object is to be used as an input to a corresponding instance of multiple instances of another task routine of the set of task routines; and
the method comprises including, by the processor, the task routine of the retrieved task routines that corresponds to the other task in the container to enable a processor incorporated into each device of the set of devices to independently perform an instance of the other task using the one of the data object blocks of the output data object stored locally within the device as an input to the instance.
28. The computer-implemented method of claim 24 , wherein:
the output data object comprises the result report; and
the method comprises:
retrieving, from each device of the set of devices, at least one data object block of the set of data object blocks of the output data object;
assembling, by the processor, the output data object from the set of data object blocks of the output data object;
storing the output data object in the at least one federated area as the result report; and
transmitting, from the processor, the output data object to the remote device as the result report.
29. The computer-implemented method of claim 21 , wherein:
a data object is received as embedded within the request; and
the method comprises:
comparing, by the computer, a size of the received data object to a predetermined threshold size to determine whether the size of the received data object is larger than the predetermined threshold size; and
in response to a determination that the size of the received data object is larger than the predetermined threshold size, performing operations comprising:
analyzing, by the processor, the flow input data set to determine whether the received data object is of a distributable form in which there is no distinct metadata structure therein, and in which data items of the received data object are organized into a single homogeneous data structure wherein the data items remain accessible from each data object block of a set of data object blocks into which the received data object may be divided independently of the other data object blocks after the division of the received data object;
in response to a determination that the received data object is not of the distributable form, converting, by the processor, the received data object block into the distributable form; and
storing the received data object as the set of data object blocks within a federated area that spans multiple devices.
30. The computer-implemented method of claim 29 , comprising, in response to a determination that the received data object is smaller than the predetermined threshold size, storing the received data object as an undivided object within a federated area within a single device.