Distributed job framework and task queue
A queue-based task management system is provided. Unlike conventional queue-based task management systems, the system described herein does not handle all tasks in the conventional manner. Rather, tasks can be associated with one of several modes, including: a queue-mode (always storing the task in the queue) and an immediate-mode (record the task, but execute it immediately if possible). The two modes may be controlled programmatically to optimize utilization of system resources. Immediate-mode is implemented by monitoring available task system resources, and executing the task immediately in-process if there are resources available; otherwise the task is delegated to the queue. This in-process execution allows the sharing of parent task resources.
1. A method comprising:
maintaining a task queue that includes a plurality of entries, wherein each entry of the plurality of entries corresponds to a not-yet-executed task;
wherein the task queue has a head and a tail;
as resources become available to execute a task that is not currently being executed, executing the task that corresponds to the entry at the head of the task queue;
when execution of a new task is requested, determining whether the new task is (a) to be executed in a queue-mode, or (b) to be executing in an immediate-mode;
if the new task is to be executed in the queue-mode, then adding an entry for the new task to the tail of the task queue;
if the new task is to be executed in the immediate-mode, then determining whether resources are available to immediately execute the new task;
if resources are available to immediately execute the new task, then executing the new task without adding any entry for the task to the task queue; and
if resources are not available to immediately execute the new task, then adding an entry for the new task to the tail of the task queue;
wherein the task queue is a first task queue associated with a first execution pool;
maintaining a second tasks queue associated with a second execution pool; and
determining whether to assign tasks to the first execution pool or the second execution pool based on priority levels assigned to the tasks;
wherein the method is performed by one or more computing devices.
2. The method of claim 1 wherein determining whether the new task is (a) to be executed in the queue-mode, or (b) to be executing in the immediate-mode comprises: determining whether the new task is a top-level task; and
if the new task is a top-level task, determining that the new task is to be executed in the queue-mode; and
if the new task is not a top-level task, determining that the new task is to be executed in the immediate-mode.
3. The method of claim 1 further comprising:
upon failure of a particular task that had a first execution mode, assigning the task a second execution mode; and
retrying execution of the particular task with the second execution mode.
4. The method of claim 3 wherein the first execution mode is the queue-mode, and the second execution mode is the immediate-mode.
5. The method of claim 1 wherein: tasks with a first priority level are assigned to the first execution pool; tasks with a second priority level are assigned to the second execution pool; the first priority level is lower than the second priority level; and
more system resources are used to process the tasks assigned to the second execution pool than to process tasks assigned to the first execution pool.
6. The method of claim 5 further comprising:
upon failure of a particular task that had the first priority level, assigning the task the second priority level; and
retrying execution of the particular task with the second priority level.
7. The method of claim 5 further comprising allowing the tasks that correspond to records in the task queue to be changed after the records have been placed in the task queue and before that tasks are executed.
8. The method of claim 1 further comprising:
when execution of a second new task is requested, determining that the second new task is to be executed in a no queue mode;
responsive to determining that the second new task is to be executed in the no queue mode, executing the second new task without queuing the second new task if resources are available; and
if resources are not available, executing the second new task without queuing the second new task as soon as the resources become available.
9. The method of claim 1 further comprising:
when execution of a second new task is requested, determining that the second new task is to be executed in a delayed queuing mode;
responsive to determining that the second new task is to be executed in the delayed queuing mode, waiting to queue the second new task until a specified amount of time has expired; and
when the specified amount of time has expired, queuing the second new task in the task queue.
10. The method of claim 1 further comprising:
when execution of a second new task is requested, determining that the second new task is to be executed in a delay mode;
responsive to determining that the second new task is to be executed in the delay mode, queuing the second new task in the task queue; and
when the second new task is at the head of the task queue, delaying execution of the second new task if a specified amount of time has not elapsed since creation of the second new task.
11. One or more non-transitory computer-readable media storing instructions which, when executed by one or more computing devices, cause:
maintaining a task queue that includes a plurality of entries, wherein each entry of the plurality of entries corresponds to a not-yet-executed task;
wherein the task queue has a head and a tail;
as resources become available to execute a task that is not currently being executed, executing the task that corresponds to the entry at the head of the task queue;
when execution of a new task is requested, determining whether the new task is (a) to be executed in a queue-mode, or (b) to be executing in an immediate-mode;
if the new task is to be executed in the queue-mode, then adding an entry for the new task to the tail of the task queue;
if the new task is to be executed in the immediate-mode, then
determining whether resources are available to immediately execute the new task;
if resources are available to immediately execute the new task, then executing the new task without adding any entry for the task to the task queue; and
if resources are not available to immediately execute the new task, then adding an entry for the new task to the tail of the task queue;
the task queue is a first task queue associated with a first execution pool;
maintaining a second tasks queue associated with a second execution pool; and
determining whether to assign tasks to the first execution pool or the second execution pool based on priority levels assigned to the tasks.
12. The one or more non-transitory computer-readable media of claim 11 wherein determining whether the new task is (a) to be executed in the queue-mode, or (b) to be executing in the immediate-mode comprises: determining whether the new task is a top-level task; and
if the new task is a top-level task, determining that the new task is to be executed in the queue-mode; and
if the new task is not a top-level task, determining that the new task is to be executed in the immediate-mode.
13. The one or more non-transitory computer-readable media of claim 11 further comprising instructions for:
upon failure of a particular task that had a first execution mode, assigning the task a second execution mode; and
retrying execution of the particular task with the second execution mode.
14. The one or more non-transitory computer-readable media of claim 13 wherein the first execution mode is the queue-mode, and the second execution mode is the immediate-mode.
15. The one or more non-transitory computer-readable media of claim 11 wherein:
tasks with a first priority level are assigned to the first execution pool;
tasks with a second priority level are assigned to the second execution pool; the first priority level is lower than the second priority level; and
more system resources are used to process tasks assigned to the second execution pool than to process the tasks assigned to the first execution pool.
16. The one or more non-transitory computer-readable media of claim 15 further comprising instructions for:
upon failure of a particular task that had the first priority level, assigning the task the second priority level; and
retrying execution of the particular task with the second priority level.
17. The one or more non-transitory computer-readable media of claim 15 further comprising instructions for allowing the tasks that correspond to records in the task queue to be changed after the records have been placed in the task queue and before that tasks are executed.
18. The one or more non-transitory computer-readable media of claim 11 further comprising instructions for:
when execution of a second new task is requested, determining that the second new task is to be executed in a no queue mode;
responsive to determining that the second new task is to be executed in the no queue mode, executing the second new task without queuing the second new task if resources are available; and
if resources are not available, executing the second new task without queuing the second new task as soon as resources become available.
19. The one or more non-transitory computer-readable media of claim 11 further comprising instructions for:
when execution of a second new task is requested, determining that the second new task is to be executed in a delayed queuing mode;
responsive to determining that the second new task is to be executed in the delayed queuing mode, waiting to queue the second new task until a specified amount of time has expired; and
when the specified amount of time has expired, queuing the second new task in the task queue.
20. The one or more non-transitory computer-readable media of claim 11 further comprising:
when execution of a second new task is requested, determining that the second new task is to be executed in a delay mode;
responsive to determining that the second new task is to be executed in the delay mode, queuing the second new task in the task queue; and
when the second new task is at the head of the task queue, delaying execution of the second new task if a specified amount of time has not elapsed since creation of the second new task.