Per task routine distributed resolver
An apparatus includes a processor to: use an identifier of a requesting device or operator thereof to identify federated area(s) to which access is authorized; based on data dependencies among a set of tasks of a job flow, derive an order of performance specifying the first task to be performed; store, within a task queue, a task routine execution request message including an identifier associated with the first task, and federated area identifier(s) of the identified federated area(s); within a resolver container, in response to storage of the task routine execution request message, use the identifier associated with the first task and identifier(s) of the federated area(s) to identify one in which a first task routine is stored; within a task container, execute the first task routine to perform the first task; and upon completion of the job flow, transmit an indication of completion to the requesting device.
1. An apparatus comprising at least one processor and a storage to store instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
receive, at the at least one processor and from a requesting device via a network, a request to perform a job flow, wherein:
the job flow is defined in a job flow definition that specifies a set of tasks to be performed via execution of a corresponding set of task routines during a performance of the job flow, and that specifies data dependencies among the set of tasks;
the job flow definition is stored among multiple job flow definitions within at least one federated area;
the set of task routines is stored among multiple task routines within the at least one federated area; and
the at least one federated area is maintained within at least one storage device;
retrieve the job flow definition from among the at least one federated area;
use an identifier of the requesting device or of an operator of the requesting device to identify a subset of the at least one federated area to which access is authorized;
based on the data dependencies among the set of tasks, derive an order of performance of the set of tasks that specifies at least a first task and a second task of the set of tasks to be performed;
store, within a task queue, a first task routine execution request message comprising an identifier associated with the first task, and at least one federated area identifier of the subset of the at least one federated area;
within a first resolver container of a first task pod, in response to the storage of the first task routine execution request message within the task queue, perform operations comprising:
use the identifier associated with the first task and the at least one federated area identifier to identify a first federated area within the subset of the at least one federated area in which a first task routine of the set of task routines is stored; and
retrieve the first task routine from the first federated area of the subset;
within a first task container of the first task pod, execute instructions of the first task routine to cause the at least one processor to perform the first task; and
in response to storage, within the task queue, of at least a first task completion message indicative of completion of execution of at least the first task routine, transmit an indication of completion of the job flow to the requesting device via the network.
2. The apparatus of claim 1 , wherein the at least one processor is caused to perform operations comprising:
store, within the task queue, a second task routine execution request message comprising an identifier associated with the second task, and the at least one federated area identifier;
within a second resolver container of a second task pod, in response to the storage of the second task routine execution request message within the task queue, perform operations comprising:
use the identifier associated with the second task and the at least one federated area identifier to identify a second federated area within the subset of the at least one federated area in which a second task routine of the set of task routines is stored; and
retrieve the second task routine from the second federated area of the subset; and
within a second task container of the second task pod, execute instructions of the second task routine to cause the at least one processor to perform the second task.
3. The apparatus of claim 1 , wherein:
the request includes an identifier of a data object to be used as an input to the first task;
the first task routine execution request message comprises the identifier of the data object; and
the at least one processor is caused to perform operations comprising:
within the first resolver container of the first task pod, perform operations comprising:
use the identifier of the first data object and the at least one federated area identifier to identify a federated area within the subset of the at least one federated area in which the data object is stored; and
retrieve the data object from the identified federated area of the subset; and
within the first task container, use the data object as an input in the execution of instructions of the first task routine.
4. The apparatus of claim 1 , wherein:
the request includes an identifier associated with the job flow;
the first task routine execution request message comprises a portion of the job flow definition that comprises the identifier associated with the first task; and
the at least one processor is caused to perform operations comprising:
use the identifier associated with the job flow to perform the retrieval of the job flow definition; and
within the first resolver container of the first task pod, perform operations comprising:
use the identifier associated with the first task and the at least one federated area identifier to identify, as the first federated area, a federated area within the subset of the at least one federated area that stores a most recent version of the first task routine available within the subset; and
retrieve the most recent version of the first task routine from the first federated area.
5. The apparatus of claim 1 , wherein:
the request includes an identifier associated with a past performance of the job flow;
the first task routine execution request message comprises an identifier of a version of the first task routine that was executed to perform the first task during the past performance of the job flow; and
the at least one processor is caused to perform operations comprising:
use the identifier associated with the past performance of the job flow to retrieve an instance log that specifies versions of task routines executed during the past performance;
retrieve the identifier of the version of the first task routine executed during the past performance from the instance log; and
within the first resolver container of the first task pod, perform operations comprising:
use the identifier of the version of the first task routine executed during the past performance and the at least one federated area identifier to identify, as the first federated area, a federated area within the subset of the at least one federated area that stores the version of the first task routine executed during the past performance; and
retrieve the version of the first task routine executed during the past performance from the first federated area.
6. The apparatus of claim 1 , wherein the at least one processor is caused to perform operations comprising:
store, within a job queue, a job performance request message comprising the job flow definition and the at least one federated area identifier of the subset of the at least one federated area;
within a performance container, execute instructions of an instance of a performance routine to cause the at least one processor to, in response to the storage of the job performance request message within the job queue, perform operations comprising:
perform the derivation of the order of performance of the set of tasks;
perform the storage of the first task routine execution request message within the task queue;
monitor the task queue for the first task completion message indicative of completion of execution of the first task routine; and
in response to storage of at least the first task completion message within the task queue, store a job completion message indicative of completion of the job flow within the job queue; and
perform the transmission of the indication of completion of the job flow to the requesting device via the network in response to the storage of the job completion message within the job queue.
7. The apparatus of claim 6 , wherein, within the performance container, the at least one processor is caused to perform operations comprising:
store a second task routine execution request message within the task queue;
monitor the task queue for a second task completion message indicative of completion of execution of a second task routine that causes the at least one processor to perform the second task; and
in response to storage of at least the first task completion message and the second task completion message within the task queue, store the job completion message within the job queue.
8. The apparatus of claim 6 , wherein the at least one processor is caused to, within a portal container, execute instructions of an instance of a portal routine to cause the at least one processor to, in response to receiving the request, perform operations comprising:
use the identifier of the requesting device or of the operator of the requesting device to access corresponding account information comprising the at least one federated area identifier of the subset of the at least one federated area;
use the at least one federated area identifier to identify a federated area within the subset in which the job flow definition is stored; and
perform the retrieval of the job flow definition from the identified federated area of the subset.
9. The apparatus of claim 1 , wherein the at least one processor is caused to perform operations comprising:
execute instructions of a resource allocation routine to dynamically allocate a plurality of task pods to support execution of a plurality of task routines at least partially in parallel to support performing a plurality of job flows at least partially in parallel based on availability of at least one of processing resources or storage resources, and based on an environment variable indicating a maximum quantity of task pods to be allocated;
based on the order of performance of the set of tasks, derive a quantity of task pods needed to support the performance of the job flow; and
provide, to the resource allocation routine, an indication of the quantity of task pods needed to support the performance of the job flow.
10. The apparatus of claim 9 , wherein:
the plurality of task pods comprises multiple types of task pod to support the execution of task routines written in any of multiple different programming languages;
each type of task pod of the multiple types of task pod supports the execution of a task routine written in a different programming language of the multiple different programming languages;
the first task routine is written in a selected programming language of the plurality of programming languages;
the environment variable indicating the maximum quantity of task pods to be allocated specifies a maximum quantity of task pods of a type that support execution of a task routine written in the selected programming language;
the quantity of task pods needed to support the performance of the job flow specifies a quantity of task pods of the type that support execution of a task routine written in the selected programming language; and
in executing instructions of the resource allocation routine, the at least one processor is caused to instantiate each task pod to include an environment variable indicating the type of the task pod to enable a routine executed within the task pod to determine which programming language of the plurality of programming languages is to be supported for the execution of a task routine within the task pod.
11. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause at least one processor to perform operations comprising:
receive, at the at least one processor and from a requesting device via a network, a request to perform a job flow, wherein:
the job flow is defined in a job flow definition that specifies a set of tasks to be performed via execution of a corresponding set of task routines during a performance of the job flow, and that specifies data dependencies among the set of tasks;
the job flow definition is stored among multiple job flow definitions within at least one federated area;
the set of task routines is stored among multiple task routines within the at least one federated area; and
the at least one federated area is maintained within at least one storage device;
retrieve the job flow definition from among the at least one federated area;
use an identifier of the requesting device or of an operator of the requesting device to identify a subset of the at least one federated area to which access is authorized;
based on the data dependencies among the set of tasks, derive an order of performance of the set of tasks that specifies at least a first task and a second task of the set of tasks to be performed;
store, within a task queue, a first task routine execution request message comprising an identifier associated with the first task, and at least one federated area identifier of the subset of the at least one federated area;
within a first resolver container of a first task pod, in response to the storage of the first task routine execution request message within the task queue, perform operations comprising:
use the identifier associated with the first task and the at least one federated area identifier to identify a first federated area within the subset of the at least one federated area in which a first task routine of the set of task routines is stored; and
retrieve the first task routine from the first federated area of the subset;
within a first task container of the first task pod, execute instructions of the first task routine to cause the at least one processor to perform the first task; and
in response to storage, within the task queue, of at least a first task completion message indicative of completion of execution of at least the first task routine, transmit an indication of completion of the job flow to the requesting device via the network.
12. The computer-program product of claim 11 , wherein the at least one processor is caused to perform operations comprising:
store, within the task queue, a second task routine execution request message comprising an identifier associated with the second task, and the at least one federated area identifier;
within a second resolver container of a second task pod, in response to the storage of the second task routine execution request message within the task queue, perform operations comprising:
use the identifier associated with the second task and the at least one federated area identifier to identify a second federated area within the subset of the at least one federated area in which a second task routine of the set of task routines is stored; and
retrieve the second task routine from the second federated area of the subset; and
within a second task container of the second task pod, execute instructions of the second task routine to cause the at least one processor to perform the second task.
13. The computer-program product of claim 11 , wherein:
the request includes an identifier of a data object to be used as an input to the first task;
the first task routine execution request message comprises the identifier of the data object; and
the at least one processor is caused to perform operations comprising:
within the first resolver container of the first task pod, perform operations comprising:
use the identifier of the first data object and the at least one federated area identifier to identify a federated area within the subset of the at least one federated area in which the data object is stored; and
retrieve the data object from the identified federated area of the subset; and
within the first task container, use the data object as an input in the execution of instructions of the first task routine.
14. The computer-program product of claim 11 , wherein:
the request includes an identifier associated with the job flow;
the first task routine execution request message comprises a portion of the job flow definition that comprises the identifier associated with the first task; and
the at least one processor is caused to perform operations comprising:
use the identifier associated with the job flow to perform the retrieval of the job flow definition; and
within the first resolver container of the first task pod, perform operations comprising:
use the identifier associated with the first task and the at least one federated area identifier to identify, as the first federated area, a federated area within the subset of the at least one federated area that stores a most recent version of the first task routine available within the subset; and
retrieve the most recent version of the first task routine from the first federated area.
15. The computer-program product of claim 11 , wherein:
the request includes an identifier associated with a past performance of the job flow;
the first task routine execution request message comprises an identifier of a version of the first task routine that was executed to perform the first task during the past performance of the job flow; and
the at least one processor is caused to perform operations comprising:
use the identifier associated with the past performance of the job flow to retrieve an instance log that specifies versions of task routines executed during the past performance;
retrieve the identifier of the version of the first task routine executed during the past performance from the instance log; and
within the first resolver container of the first task pod, perform operations comprising:
use the identifier of the version of the first task routine executed during the past performance and the at least one federated area identifier to identify, as the first federated area, a federated area within the subset of the at least one federated area that stores the version of the first task routine executed during the past performance; and
retrieve the version of the first task routine executed during the past performance from the first federated area.
16. The computer-program product of claim 11 , wherein the at least one processor is caused to perform operations comprising:
store, within a job queue, a job performance request message comprising the job flow definition and the at least one federated area identifier of the subset of the at least one federated area;
within a performance container, execute instructions of an instance of a performance routine to cause the at least one processor to, in response to the storage of the job performance request message within the job queue, perform operations comprising:
perform the derivation of the order of performance of the set of tasks;
perform the storage of the first task routine execution request message within the task queue;
monitor the task queue for the first task completion message indicative of completion of execution of the first task routine; and
in response to storage of at least the first task completion message within the task queue, store a job completion message indicative of completion of the job flow within the job queue; and
perform the transmission of the indication of completion of the job flow to the requesting device via the network in response to the storage of the job completion message within the job queue.
17. The computer-program product of claim 16 , wherein, within the performance container, the at least one processor is caused to perform operations comprising:
store a second task routine execution request message within the task queue;
monitor the task queue for a second task completion message indicative of completion of execution of a second task routine that causes the at least one processor to perform the second task; and
in response to storage of at least the first task completion message and the second task completion message within the task queue, store the job completion message within the job queue.
18. The computer-program product of claim 16 , wherein the at least one processor is caused to, within a portal container, execute instructions of an instance of a portal routine to cause the at least one processor to, in response to receiving the request, perform operations comprising:
use the identifier of the requesting device or of the operator of the requesting device to access corresponding account information comprising the at least one federated area identifier of the subset of the at least one federated area;
use the at least one federated area identifier to identify a federated area within the subset in which the job flow definition is stored; and
perform the retrieval of the job flow definition from the identified federated area of the subset.
19. The computer-program product of claim 11 , wherein the at least one processor is caused to perform operations comprising:
execute instructions of a resource allocation routine to dynamically allocate a plurality of task pods to support execution of a plurality of task routines at least partially in parallel to support performing a plurality of job flows at least partially in parallel based on availability of at least one of processing resources or storage resources, and based on an environment variable indicating a maximum quantity of task pods to be allocated;
based on the order of performance of the set of tasks, derive a quantity of task pods needed to support the performance of the job flow; and
provide, to the resource allocation routine, an indication of the quantity of task pods needed to support the performance of the job flow.
20. The computer-program product of claim 19 , wherein:
the plurality of task pods comprises multiple types of task pod to support the execution of task routines written in any of multiple different programming languages;
each type of task pod of the multiple types of task pod supports the execution of a task routine written in a different programming language of the multiple different programming languages;
the first task routine is written in a selected programming language of the plurality of programming languages;
the environment variable indicating the maximum quantity of task pods to be allocated specifies a maximum quantity of task pods of a type that support execution of a task routine written in the selected programming language;
the quantity of task pods needed to support the performance of the job flow specifies a quantity of task pods of the type that support execution of a task routine written in the selected programming language; and
in executing instructions of the resource allocation routine, the at least one processor is caused to instantiate each task pod to include an environment variable indicating the type of the task pod to enable a routine executed within the task pod to determine which programming language of the plurality of programming languages is to be supported for the execution of a task routine within the task pod.
21. A computer-implemented method comprising:
receiving, by at least one processor, and from a requesting device via a network, a request to perform a job flow, wherein:
the job flow is defined in a job flow definition that specifies a set of tasks to be performed via execution of a corresponding set of task routines during a performance of the job flow, and that specifies data dependencies among the set of tasks;
the job flow definition is stored among multiple job flow definitions within at least one federated area;
the set of task routines is stored among multiple task routines within the at least one federated area; and
the at least one federated area is maintained within at least one storage device;
retrieving the job flow definition from among the at least one federated area;
using, by the at least one processor, an identifier of the requesting device or of an operator of the requesting device to identify a subset of the at least one federated area to which access is authorized;
based on the data dependencies among the set of tasks, deriving, by the at least one processor, an order of performance of the set of tasks that specifies at least a first task and a second task of the set of tasks to be performed;
storing, within a task queue, a first task routine execution request message comprising an identifier associated with the first task, and at least one federated area identifier of the subset of the at least one federated area;
within a first resolver container of a first task pod, in response to the storage of the first task routine execution request message within the task queue, performing operations comprising:
using, by the at least one processor, the identifier associated with the first task and the at least one federated area identifier to identify a first federated area within the subset of the at least one federated area in which a first task routine of the set of task routines is stored; and
retrieving the first task routine from the first federated area of the subset;
within a first task container of the first task pod, executing, by the at least one processor, instructions of the first task routine to cause the at least one processor to perform the first task; and
in response to storage, within the task queue, of at least a first task completion message indicative of completion of execution of at least the first task routine, transmitting, from the at least one processor, an indication of completion of the job flow to the requesting device via the network.
22. The computer-implemented method of claim 21 , comprising:
storing, within the task queue, a second task routine execution request message comprising an identifier associated with the second task, and the at least one federated area identifier;
within a second resolver container of a second task pod, in response to the storage of the second task routine execution request message within the task queue, performing operations comprising:
using the identifier associated with the second task and the at least one federated area identifier to identify a second federated area within the subset of the at least one federated area in which a second task routine of the set of task routines is stored; and
retrieving the second task routine from the second federated area of the subset; and
within a second task container of the second task pod, executing, by the at least one processor, instructions of the second task routine to cause the at least one processor to perform the second task.
23. The computer-implemented method of claim 21 , wherein:
the request includes an identifier of a data object to be used as an input to the first task;
the first task routine execution request message comprises the identifier of the data object; and
the method comprises:
within the first resolver container of the first task pod, performing operations comprising:
using the identifier of the first data object and the at least one federated area identifier to identify a federated area within the subset of the at least one federated area in which the data object is stored; and
retrieving the data object from the identified federated area of the subset; and
within the first task container, use the data object as an input in the execution of instructions of the first task routine.
24. The computer-implemented method of claim 21 , wherein:
the request includes an identifier associated with the job flow;
the first task routine execution request message comprises a portion of the job flow definition that comprises the identifier associated with the first task; and
the method comprises:
using the identifier associated with the job flow to perform the retrieval of the job flow definition; and
within the first resolver container of the first task pod, performing operations comprising:
using the identifier associated with the first task and the at least one federated area identifier to identify, as the first federated area, a federated area within the subset of the at least one federated area that stores a most recent version of the first task routine available within the subset; and
retrieving the most recent version of the first task routine from the first federated area.
25. The computer-implemented method of claim 21 , wherein:
the request includes an identifier associated with a past performance of the job flow;
the first task routine execution request message comprises an identifier of a version of the first task routine that was executed to perform the first task during the past performance of the job flow; and
the method comprises:
using the identifier associated with the past performance of the job flow to retrieve an instance log that specifies versions of task routines executed during the past performance;
retrieving the identifier of the version of the first task routine executed during the past performance from the instance log; and
within the first resolver container of the first task pod, performing operations comprising:
using the identifier of the version of the first task routine executed during the past performance and the at least one federated area identifier to identify, as the first federated area, a federated area within the subset of the at least one federated area that stores the version of the first task routine executed during the past performance; and
retrieving the version of the first task routine executed during the past performance from the first federated area.
26. The computer-implemented method of claim 21 , comprising:
storing, within a job queue, a job performance request message comprising the job flow definition and the at least one federated area identifier of the subset of the at least one federated area;
within a performance container, executing, by the at least one processor, instructions of an instance of a performance routine to cause the at least one processor to, in response to the storage of the job performance request message within the job queue, perform operations comprising:
perform the derivation of the order of performance of the set of tasks;
perform the storage of the first task routine execution request message within the task queue;
monitor the task queue for the first task completion message indicative of completion of execution of the first task routine; and
in response to storage of at least the first task completion message within the task queue, store a job completion message indicative of completion of the job flow within the job queue; and
performing the transmission of the indication of completion of the job flow to the requesting device via the network in response to the storage of the job completion message within the job queue.
27. The computer-implemented method of claim 26 , comprising, within the performance container, performing operations comprising:
storing a second task routine execution request message within the task queue;
monitoring the task queue for a second task completion message indicative of completion of execution of a second task routine that causes the at least one processor to perform the second task; and
in response to storage of at least the first task completion message and the second task completion message within the task queue, storing the job completion message within the job queue.
28. The computer-implemented method of claim 26 , comprising, within a portal container, executing, by the at least one processor, instructions of an instance of a portal routine to cause the at least one processor to, in response to receiving the request, perform operations comprising:
use the identifier of the requesting device or of the operator of the requesting device to access corresponding account information comprising the at least one federated area identifier of the subset of the at least one federated area;
use the at least one federated area identifier to identify a federated area within the subset in which the job flow definition is stored; and
perform the retrieval of the job flow definition from the identified federated area of the subset.
29. The computer-implemented method of claim 21 , comprising:
executing, by the at least one processor, instructions of a resource allocation routine to dynamically allocate a plurality of task pods to support execution of a plurality of task routines at least partially in parallel to support performing a plurality of job flows at least partially in parallel based on availability of at least one of processing resources or storage resources, and based on an environment variable indicating a maximum quantity of task pods to be allocated;
based on the order of performance of the set of tasks, deriving a quantity of task pods needed to support the performance of the job flow; and
providing, to the resource allocation routine, an indication of the quantity of task pods needed to support the performance of the job flow.
30. The computer-implemented method of claim 29 , wherein:
the plurality of task pods comprises multiple types of task pod to support the execution of task routines written in any of multiple different programming languages;
each type of task pod of the multiple types of task pod supports the execution of a task routine written in a different programming language of the multiple different programming languages;
the first task routine is written in a selected programming language of the plurality of programming languages;
the environment variable indicating the maximum quantity of task pods to be allocated specifies a maximum quantity of task pods of a type that support execution of a task routine written in the selected programming language;
the quantity of task pods needed to support the performance of the job flow specifies a quantity of task pods of the type that support execution of a task routine written in the selected programming language; and
in executing instructions of the resource allocation routine, the at least one processor is caused to instantiate each task pod to include an environment variable indicating the type of the task pod to enable a routine executed within the task pod to determine which programming language of the plurality of programming languages is to be supported for the execution of a task routine within the task pod.