Granted Patent B2

US 11,204,809 · App. 17/308,355 · Granted Dec 21, 2021

Inventors: Henry Gabriel Victor Bequet (Cary, NC); Ronald Earl Stogner (Cary, NC); Eric Jian Yang (Morrisville, NC); Qing Gong (Cary, NC); Partha Dutta (Morrisville, NC); Kais Arfaoui (Raleigh, NC)

Assignee: SAS INSTITUTE INC.

G06F9/5038G06F9/46G06F9/4881G06F9/54G06F9/544G06F9/546G06F9/455

Abstract

Claims (216)

1. An apparatus comprising at least one processor and a storage to store instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:

receive, at the at least one processor and from a requesting device via a network, a request to perform a job flow, wherein:

the job flow is defined in a job flow definition that specifies a set of tasks to be performed via execution of a corresponding set of task routines during a performance of the job flow, and that specifies data dependencies among the set of tasks;

the job flow definition is stored among multiple job flow definitions within at least one federated area;

the set of task routines is stored among multiple task routines within the at least one federated area; and

the at least one federated area is maintained within at least one storage device;

retrieve the job flow definition from among the multiple job flow definitions in the at least one federated area;

within a performance container, execute instructions of an instance of a performance routine to cause the at least one processor to perform operations comprising:

based on the data dependencies among the set of tasks, identify a first task, a second task and a third task of the set of tasks to be performed, wherein:

the second task is to be performed sequentially after the first task is performed;

the first task is to output a data object that is to become an input to the second task; and

the third task is to be performed at least partially in parallel with at least one of the first task or the second task;

store, within a task queue, at least one task routine execution request message that conveys at least an identifier associated with the first task, and that conveys an indication that the data object is to be exchanged between the first task and the second task through a shared memory space; and

store, within the task queue, another task routine execution request message comprising an identifier associated with the third task to enable execution of a third task routine within a second task container to cause a performance of the third task at least partially in parallel with at least one of the performance of the first task or the performance of the second task;

within a first task container, in response to storage of the at least one task routine execution request message within the task queue, the at least one processor is caused to perform operations comprising:

execute instructions of a first task routine to cause a performance of the first task;

execute instructions of a second task routine to cause a performance of the second task; and

instantiate the shared memory space through which the data object is exchanged from the first task routine to the second task routine to be accessible to the first task routine during execution of the instructions of the first task routine, and to be accessible to the second task routine during execution of the instructions of the second task routine; and

upon completion of the set of tasks of the job flow, transmit an indication of completion of the job flow to the requesting device via the network.

2. The apparatus of claim 1 , wherein, within the first task container, and in response to the storage of the at least one task routine execution request message within the task queue, the at least one processor is caused to perform operations comprising:

store, within the task queue, at least one task in progress message indicative of execution of at least the first task routine being in progress;

during execution of the instructions of the first task routine, generate the data object within the shared memory space;

during execution of the instructions of the second task routine, perform at least one operation on the data object in situ within the shared memory space as part of using the data object as an input; and

store, within the task queue, at least one task completion message indicative of completion of execution of at least the first task routine.

3. The apparatus of claim 2 , wherein the at least one processor is caused to perform operations comprising:

store, within a job queue, a job performance request message comprising the job flow definition;

within the performance container, perform operations comprising:

in response to the storage of the job performance request message within the job queue, identify the first task and the second task to be performed, and store the at least one task routine execution message within the task queue; and

in response to the storage of the at least one task completion message within the task queue, store a job completion message indicative of completion of the job flow within the job queue; and

in response to the storage of the job completion message within the job queue, perform the transmission the indication of completion of the job flow to the requesting device via the network.

4. The apparatus of claim 2 , wherein:

a set of data objects is stored within the at least one federated area in a first form that is compatible with executable instructions written in a primary programming language, but not with executable instructions written in a secondary programming language;

an input data object retrieved from the at least one federated area must be converted from the first form and into a second form to become usable as an input to a task routine written in the secondary programming language;

an output data object output by a task routine written in the secondary programming language is required to be converted from the second form and into the first form before being stored within the at least one federated area to become usable as an input to a task routine written in the primary programming language;

the first task routine and the second task routine are written in the secondary programming language; and

within the first task container, the at least one processor is caused to perform operations comprising:

generate a copy of the data object generated by the first task routine that is converted from the second form and into the first form of the data object; and

store the first form of the data object within the at least one federated area at least partially in parallel with the use of the data object as an input to the second task routine.

5. The apparatus of claim 1 , wherein:

storage, within the task queue, of the at least one task routine execution request message comprises:

storage, within the task queue of a first task routine execution request message comprising the identifier associated with the first task and the indication that a data object is to be exchanged between the first task and the second task; and

storage, within the task queue of a second task routine execution request message comprising the identifier associated with the second task;

the at least one processor is caused to perform operations comprising:

execute instructions of a resource allocation routine to dynamically allocate a plurality of task containers to support execution of a plurality of task routines at least partially in parallel to support performing a plurality of job flows at least partially in parallel based on availability of at least one of processing resources or storage resources; and

within the performance container, provide, to the resource allocation routine, an indication of a reduced quantity of task containers needed to support the performance of the job flow to increase a likelihood that the first task routine and the second task routine will be executed within the first task container; and

within the first task container, the at least one processor is caused to perform operations comprising:

 in response to the storage, within the task queue, of the first task routine execution request message, perform operations comprising:

perform the instantiation of the shared memory space;

perform the execution of the instructions of the first task routine to perform the first task, including generation of the data object within the shared memory space; and

store, within the task queue, a first task completion message; and

 in response to the storage, within the task queue, of the second task routine execution request message, perform operations comprising:

perform the execution of the instructions of the second task routine, including the performance of the at least one operation on the data object in situ within the shared memory space; and

store, within the task queue, a second task completion message.

6. The apparatus of claim 5 , wherein:

the dynamic allocation of a plurality of task containers comprises a dynamic allocation of a plurality of task pods;

the plurality of task pods comprises multiple types of task pod;

each type of task pod of the multiple types of task pod supports execution of a task routine written in a corresponding different programming language of a plurality of programming languages;

the first task routine and the second task routine are written in a selected programming language of the plurality of programming languages; and

the reduced quantity of task containers needed to support the performance of the job flow specifies a quantity of task pods of the type that support execution of a task routine written in the selected programming language.

7. The apparatus of claim 5 , wherein:

the dynamic allocation of a plurality of task containers comprises a dynamic allocation of a plurality of task pods;

the dynamic allocation of the plurality of task pods is also based on an environment variable indicating a maximum quantity of task pods to be allocated; and

each task pod is instantiated to include an environment variable indicating a type of the task pod of multiple types of task pod to enable a routine executed within the task pod to configure the task pod to be of the type specified.

8. The apparatus of claim 1 , wherein:

storage, within the task queue, of the at least one task routine execution request message comprises storage, within the task queue of a single task routine execution request message comprising the identifier associated with the first task, the identifier associated with the second task and the indication that a data object is to be exchanged between the first task and the second task; and

within the first task container, in response to the storage, within the task queue, of the single task routine execution request message, the at least one processor is caused to perform operations comprising:

perform the instantiation of the shared memory space;

perform the execution of the instructions of the first task routine to perform the first task, including generation of the data object within the shared memory space;

store, within the task queue, a first task completion message;

perform the execution of the instructions of the second task routine, including the performance of the at least one operation on the data object in situ within the shared memory space; and

store, within the task queue, a second task completion message.

9. The apparatus of claim 8 , wherein the single task routine execution message comprises a portion of the job flow definition that includes the identifier associated with the first task and the identifier associated with the second task.

10. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause at least one processor to perform operations comprising:

receive, at the at least one processor and from a requesting device via a network, a request to perform a job flow, wherein:

the job flow is defined in a job flow definition that specifies a set of tasks to be performed via execution of a corresponding set of task routines during a performance of the job flow, and that specifies data dependencies among the set of tasks;

the job flow definition is stored among multiple job flow definitions within at least one federated area;

the set of task routines is stored among multiple task routines within the at least one federated area; and

the at least one federated area is maintained within at least one storage device;

retrieve the job flow definition from among the multiple job flow definitions in the at least one federated area;

within a performance container, execute instructions of an instance of a performance routine to cause the at least one processor to perform operations comprising:

based on the data dependencies among the set of tasks, identify a first task a second task and a third task of the set of tasks to be performed, wherein:

the second task is to be performed sequentially after the first task is performed;

the first task is to output a data object that is to become an input to the second task; and

the third task is to be performed at least partially in parallel with at least one of the first task or the second task;

store, within a task queue, at least one task routine execution request message that conveys at least an identifier associated with the first task, and that conveys an indication that the data object is to be exchanged between the first task and the second task through a shared memory space; and

store, within the task queue, another task routine execution request message comprising an identifier associated with the third task to enable execution of a third task routine within a second task container to cause a performance of the third task at least partially in parallel with at least one of the performance of the first task or the performance of the second task;

within a first task container, in response to storage of the at least one task routine execution request message within the task queue, the at least one processor is caused to perform operations comprising:

execute instructions of a first task routine to cause a performance of the first task;

execute instructions of a second task routine to cause a performance of the second task; and

instantiate the shared memory space through which the data object is exchanged from the first task routine to the second task routine to be accessible to the first task routine during execution of the instructions of the first task routine, and to be accessible to the second task routine during execution of the instructions of the second task routine; and

upon completion of the set of tasks of the job flow, transmit an indication of completion of the job flow to the requesting device via the network.

11. The computer-program product of claim 10 , wherein, within the first task container, and in response to the storage of the at least one task routine execution request message within the task queue, the at least one processor is caused to perform operations comprising:

store, within the task queue, at least one task in progress message indicative of execution of at least the first task routine being in progress;

during execution of the instructions of the first task routine, generate the data object within the shared memory space;

during execution of the instructions of the second task routine, perform at least one operation on the data object in situ within the shared memory space as part of using the data object as an input; and

store, within the task queue, at least one task completion message indicative of completion of execution of at least the first task routine.

12. The computer-program product of claim 11 , wherein the at least one processor is caused to perform operations comprising:

store, within a job queue, a job performance request message comprising the job flow definition;

within the performance container, perform operations comprising:

in response to the storage of the job performance request message within the job queue, identify the first task and the second task to be performed, and store the at least one task routine execution message within the task queue; and

in response to the storage of the at least one task completion message within the task queue, store a job completion message indicative of completion of the job flow within the job queue; and

in response to the storage of the job completion message within the job queue, perform the transmission the indication of completion of the job flow to the requesting device via the network.

13. The computer-program product of claim 11 , wherein:

a set of data objects is stored within the at least one federated area in a first form that is compatible with executable instructions written in a primary programming language, but not with executable instructions written in a secondary programming language;

an input data object retrieved from the at least one federated area must be converted from the first form and into a second form to become usable as an input to a task routine written in the secondary programming language;

an output data object output by a task routine written in the secondary programming language is required to be converted from the second form and into the first form before being stored within the at least one federated area to become usable as an input to a task routine written in the primary programming language;

the first task routine and the second task routine are written in the secondary programming language; and

within the first task container, the at least one processor is caused to perform operations comprising:

generate a copy of the data object generated by the first task routine that is converted from the second form and into the first form of the data object; and

store the first form of the data object within the at least one federated area at least partially in parallel with the use of the data object as an input to the second task routine.

14. The computer-program product of claim 10 , wherein:

storage, within the task queue, of the at least one task routine execution request message comprises:

storage, within the task queue of a first task routine execution request message comprising the identifier associated with the first task and the indication that a data object is to be exchanged between the first task and the second task; and

storage, within the task queue of a second task routine execution request message comprising the identifier associated with the second task;

the at least one processor is caused to perform operations comprising:

execute instructions of a resource allocation routine to dynamically allocate a plurality of task containers to support execution of a plurality of task routines at least partially in parallel to support performing a plurality of job flows at least partially in parallel based on availability of at least one of processing resources or storage resources; and

within the performance container, provide, to the resource allocation routine, an indication of a reduced quantity of task containers needed to support the performance of the job flow to increase a likelihood that the first task routine and the second task routine will be executed within the first task container; and

within the first task container, the at least one processor is caused to perform operations comprising:

 in response to the storage, within the task queue, of the first task routine execution request message, perform operations comprising:

perform the instantiation of the shared memory space;

perform the execution of the instructions of the first task routine to perform the first task, including generation of the data object within the shared memory space; and

store, within the task queue, a first task completion message; and

 in response to the storage, within the task queue, of the second task routine execution request message, perform operations comprising:

perform the execution of the instructions of the second task routine, including the performance of the at least one operation on the data object in situ within the shared memory space; and

store, within the task queue, a second task completion message.

15. The computer-program product of claim 14 , wherein:

the dynamic allocation of a plurality of task containers comprises a dynamic allocation of a plurality of task pods;

the plurality of task pods comprises multiple types of task pod;

each type of task pod of the multiple types of task pod supports execution of a task routine written in a corresponding different programming language of a plurality of programming languages;

the first task routine and the second task routine are written in a selected programming language of the plurality of programming languages; and

the reduced quantity of task containers needed to support the performance of the job flow specifies a quantity of task pods of the type that support execution of a task routine written in the selected programming language.

16. The computer-program product of claim 14 , wherein:

the dynamic allocation of a plurality of task containers comprises a dynamic allocation of a plurality of task pods;

the dynamic allocation of the plurality of task pods is also based on an environment variable indicating a maximum quantity of task pods to be allocated; and

each task pod is instantiated to include an environment variable indicating a type of the task pod of multiple types of task pod to enable a routine executed within the task pod to configure the task pod to be of the type specified.

17. The computer-program product of claim 10 , wherein:

storage, within the task queue, of the at least one task routine execution request message comprises storage, within the task queue of a single task routine execution request message comprising the identifier associated with the first task, the identifier associated with the second task and the indication that a data object is to be exchanged between the first task and the second task; and

within the first task container, in response to the storage, within the task queue, of the single task routine execution request message, the at least one processor is caused to perform operations comprising:

perform the instantiation of the shared memory space;

perform the execution of the instructions of the first task routine to perform the first task, including generation of the data object within the shared memory space;

store, within the task queue, a first task completion message;

perform the execution of the instructions of the second task routine, including the performance of the at least one operation on the data object in situ within the shared memory space; and

store, within the task queue, a second task completion message.

18. The computer-program product of claim 17 , wherein the single task routine execution message comprises a portion of the job flow definition that includes the identifier associated with the first task and the identifier associated with the second task.

19. A computer-implemented method comprising:

receiving, by at the at least one processor, and from a requesting device via a network, a request to perform a job flow, wherein:

the job flow is defined in a job flow definition that specifies a set of tasks to be performed via execution of a corresponding set of task routines during a performance of the job flow, and that specifies data dependencies among the set of tasks;

the job flow definition is stored among multiple job flow definitions within at least one federated area;

the set of task routines is stored among multiple task routines within the at least one federated area; and

the at least one federated area is maintained within at least one storage device;

retrieving the job flow definition from among the multiple job flow definitions in the at least one federated area;

within a performance container, executing, by the at least one processor, instructions of an instance of a performance routine to cause the at least one processor to perform operations comprising:

based on the data dependencies among the set of tasks, identifying a first task a second task and a third task of the set of tasks to be performed, wherein:

the second task is to be performed sequentially after the first task is performed;

the first task is to output a data object that is to become an input to the second task; and

the third task is to be performed at least partially in parallel with at least one of the first task or the second task;

storing, within a task queue, at least one task routine execution request message that conveys at least an identifier associated with the first task, and that conveys an indication that the data object is to be exchanged between the first task and the second task through a shared memory space; and

storing, within the task queue, another task routine execution request message comprising an identifier associated with the third task to enable execution of a third task routine within a second task container to cause a performance of the third task at least partially in parallel with at least one of the performance of the first task or the performance of the second task;

within a first task container, in response to storage of the at least one task routine execution request message within the task queue, performing operations comprising:

executing, by the at least one processor, instructions of a first task routine to cause a performance of the first task;

executing, by the at least one processor, instructions of a second task routine to cause a performance of the second task; and

instantiating, by the at least one processor, the shared memory space through which the data object is exchanged from the first task routine to the second task routine to be accessible to the first task routine during execution of the instructions of the first task routine, and to be accessible to the second task routine during execution of the instructions of the second task routine; and

upon completion of the set of tasks of the job flow, transmitting, from the at least one processor, an indication of completion of the job flow to the requesting device via the network.

20. The computer-implemented method of claim 19 , comprising, within the first task container, and in response to the storage of the at least one task routine execution request message within the task queue, performing operations comprising:

storing, within the task queue, at least one task in progress message indicative of execution of at least the first task routine being in progress;

during execution of the instructions of the first task routine, generating, by the at least one processor, the data object within the shared memory space;

during execution of the instructions of the second task routine, performing, by the at least one processor, at least one operation on the data object in situ within the shared memory space as part of using the data object as an input; and

storing, within the task queue, at least one task completion message indicative of completion of execution of at least the first task routine.

21. The computer-implemented method of claim 20 , comprising performing operations comprising:

storing, within a job queue, a job performance request message comprising the job flow definition;

within the performance container, performing operations comprising:

in response to the storage of the job performance request message within the job queue, identifying the first task and the second task to be performed, and store the at least one task routine execution message within the task queue; and

in response to the storage of the at least one task completion message within the task queue, storing a job completion message indicative of completion of the job flow within the job queue; and

in response to the storage of the job completion message within the job queue, performing the transmission the indication of completion of the job flow to the requesting device via the network.

22. The computer-implemented method of claim 20 , wherein:

a set of data objects is stored within the at least one federated area in a first form that is compatible with executable instructions written in a primary programming language, but not with executable instructions written in a secondary programming language;

an input data object retrieved from the at least one federated area must be converted from the first form and into a second form to become usable as an input to a task routine written in the secondary programming language;

an output data object output by a task routine written in the secondary programming language is required to be converted from the second form and into the first form before being stored within the at least one federated area to become usable as an input to a task routine written in the primary programming language;

the first task routine and the second task routine are written in the secondary programming language; and

the method comprises, within the first task container, performing operations comprising:

generating, by the at least one processor, a copy of the data object generated by the first task routine that is converted from the second form and into the first form of the data object; and

storing the first form of the data object within the at least one federated area at least partially in parallel with the use of the data object as an input to the second task routine.

23. The computer-implemented method of claim 19 , wherein:

storage, within the task queue, of the at least one task routine execution request message comprises:

storage, within the task queue of a first task routine execution request message comprising the identifier associated with the first task and the indication that a data object is to be exchanged between the first task and the second task; and

storage, within the task queue of a second task routine execution request message comprising the identifier associated with the second task;

the method comprises:

executing, by the at least one processor, instructions of a resource allocation routine to dynamically allocate a plurality of task containers to support execution of a plurality of task routines at least partially in parallel to support performing a plurality of job flows at least partially in parallel based on availability of at least one of processing resources or storage resources; and

within the performance container, providing, to the resource allocation routine, an indication of a reduced quantity of task containers needed to support the performance of the job flow to increase a likelihood that the first task routine and the second task routine will be executed within the first task container; and

the method further comprises, within the first task container, performing operations comprising:

 in response to the storage, within the task queue, of the first task routine execution request message, performing operations comprising:

performing the instantiation, by the at least one processor, of the shared memory space;

performing the execution, by the at least one processor, of the instructions of the first task routine to perform the first task, including generation of the data object within the shared memory space; and

storing, within the task queue, a first task completion message; and

 in response to the storage, within the task queue, of the second task routine execution request message, perform operations comprising:

performing the execution, by the at least one processor, of the instructions of the second task routine, including the performance of the at least one operation on the data object in situ within the shared memory space; and

 storing, within the task queue, a second task completion message.

24. The computer-implemented method of claim 23 , wherein:

the dynamic allocation of a plurality of task containers comprises a dynamic allocation of a plurality of task pods;

the plurality of task pods comprises multiple types of task pod;

each type of task pod of the multiple types of task pod supports execution of a task routine written in a corresponding different programming language of a plurality of programming languages;

the first task routine and the second task routine are written in a selected programming language of the plurality of programming languages; and

the reduced quantity of task containers needed to support the performance of the job flow specifies a quantity of task pods of the type that support execution of a task routine written in the selected programming language.

25. The computer-implemented method of claim 23 , wherein:

the dynamic allocation of a plurality of task containers comprises a dynamic allocation of a plurality of task pods;

the dynamic allocation of the plurality of task pods is also based on an environment variable indicating a maximum quantity of task pods to be allocated; and

each task pod is instantiated to include an environment variable indicating a type of the task pod of multiple types of task pod to enable a routine executed within the task pod to configure the task pod to be of the type specified.

26. The computer-implemented method of claim 19 , wherein:

storage, within the task queue, of the at least one task routine execution request message comprises storage, within the task queue of a single task routine execution request message comprising the identifier associated with the first task, the identifier associated with the second task and the indication that a data object is to be exchanged between the first task and the second task; and

the method comprises, within the first task container, in response to the storage, within the task queue, of the single task routine execution request message, performing operations comprising:

performing the instantiation, by the at least one processor, of the shared memory space;

performing the execution, by the at least one processor, of the instructions of the first task routine to perform the first task, including generation of the data object within the shared memory space;

storing, within the task queue, a first task completion message;

performing the execution, by the at least one processor, of the instructions of the second task routine, including the performance of the at least one operation on the data object in situ within the shared memory space; and

storing, within the task queue, a second task completion message.

27. The computer-implemented method of claim 26 , wherein the single task routine execution message comprises a portion of the job flow definition that includes the identifier associated with the first task and the identifier associated with the second task.

Assignments (1)

Continuity (41)