Asynchronous task queue configuration in a database system

1. A method comprising:

decoding, by at least one hardware processor, an enqueue request received from a data process of a database system, the enqueue request including a task;

enqueuing the task and one or more additional tasks in an in-memory task queue, the enqueuing based on a pre-configured queuing scheme;

encoding an enqueue acknowledgment for transmission to the data process responsive to the enqueue request;

persisting the task in a storage location associated with the in-memory task queue;

responsive to a lease request received from a worker node, initiating a lease of the task to the worker node, the initiating including updating a state of the task maintained in the storage location; and

responsive to a dequeue request received from the worker node and indicating completion of the task by the worker node, dequeuing the task from the in-memory task queue.

2. The method of claim 1 , further comprising:

updating the state of the task in the storage location to a lease state, the lease state indicating the task is leased to the worker node for processing.

3. The method of claim 2 , further comprising:

initiating a lease timer for a pre-configured period, the lease timer starting at a time the lease is initiated.

4. The method of claim 3 , wherein the dequeue request is received prior to expiration of the timer.

5. The method of claim 1 , further comprising:

updating the storage location based on the completion of the task and responsive to the dequeue request received from the worker node.

6. The method of claim 1 , further comprising:

sharding the in-memory task queue into a plurality of shards; and

storing the task in a first shard of the plurality of shards.

7. The method of claim 6 , further comprising:

decoding a second enqueue request received from the data process, the second enqueue request including a second task; and

enqueuing the second task in a second shard of the plurality of shards.

8. The method of claim 7 , further comprising:

enqueuing one or more additional tasks received after the second task, wherein the one or more additional tasks are enqueued into corresponding shards of the plurality of shards in a round-robin configuration.

9. The method of claim 1 , further comprising:

enqueuing a second lease request received from a second worker node for a second task in the in-memory task queue; and

responsive to the enqueuing of the second lease request:

initiating a second lease of the second task to the second worker node;

initiating a lease timer for a pre-configured period, the lease timer starting a time the second lease is initiated; and

changing a state of the second task from a ready state to a leased state.

10. The method of claim 9 , further comprising:

detecting the lease timer has expired;

performing a revocation of the second lease; and

changing the state of the second task from the leased state to the ready state.

11. A system comprising:

at least one hardware processor; and

at least one memory storing instructions that cause the at least one hardware processor to perform operations comprising:

decoding an enqueue request received from a data process of a database system, the enqueue request including a task;

enqueuing the task and one or more additional tasks in an in-memory task queue, the enqueuing based on a pre-configured queuing scheme;

encoding an enqueue acknowledgment for transmission to the data process responsive to the enqueue request;

persisting the task in a storage location associated with the in-memory task queue;

responsive to a lease request received from a worker node, initiating a lease of the task to the worker node, the initiating including updating a state of the task maintained in the storage location; and

responsive to a dequeue request received from the worker node and indicating completion of the task by the worker node, dequeuing the task from the in-memory task queue.

12. The system of claim 11 , the operations further comprising:

updating the state of the task in the storage location to a lease state, the lease state indicating the task is leased to the worker node for processing.

13. The system of claim 12 , the operations further comprising:

initiating a lease timer for a pre-configured period, the lease timer starting a time the lease is initiated.

14. The system of claim 13 , wherein the dequeue request is received prior to expiration of the timer.

15. The system of claim 11 , the operations further comprising:

updating the storage location based on the completion of the task and responsive to the dequeue request received from the worker node.

16. The system of claim 11 , the operations further comprising:

sharding the in-memory task queue into a plurality of shards; and

storing the task in a first shard of the plurality of shards.

17. The system of claim 16 , the operations further comprising:

decoding a second enqueue request received from the data process, the second enqueue request including a second task; and

enqueuing the second task in a second shard of the plurality of shards.

18. The system of claim 17 , the operations further comprising:

enqueuing one or more additional tasks received after the second task, wherein the one or more additional tasks are enqueued into corresponding shards of the plurality of shards in a round-robin configuration.

19. The system of claim 11 , the operations further comprising:

enqueuing a second lease request received from a second worker node for a second task in the in-memory task queue; and

responsive to the enqueuing of the second lease request:

initiating a second lease of the second task to the second worker node;

initiating a lease timer for a pre-configured period, the lease timer starting a time the second lease is initiated; and

changing a state of the second task from a ready state to a leased state.

20. The system of claim 19 , the operations further comprising:

detecting the lease timer has expired;

performing a revocation of the second lease; and

changing the state of the second task from the leased state to the ready state.

21. A computer-storage medium comprising instructions that, when executed by one or more processors of a machine, configure the machine to perform operations comprising:

decoding an enqueue request received from a data process of a database system, the enqueue request including a task;

enqueuing the task and one or more additional tasks in an in-memory task queue, the enqueuing based on a pre-configured queuing scheme;

encoding an enqueue acknowledgment for transmission to the data process responsive to the enqueue request;

persisting the task in a storage location associated with the in-memory task queue;

responsive to a lease request received from a worker node, initiating a lease of the task to the worker node, the initiating including updating a state of the task maintained in the storage location; and

responsive to a dequeue request received from the worker node and indicating completion of the task by the worker node, dequeuing the task from the in-memory task queue.

22. The computer-storage medium of claim 21 , the operations further comprising:

updating the state of the task in the storage location to a lease state, the lease state indicating the task is leased to the worker node for processing.

23. The computer-storage medium of claim 22 , the operations further comprising:

initiating a lease timer for a pre-configured period, the lease timer starting a time the lease is initiated.

24. The computer-storage medium of claim 23 , wherein the dequeue request is received prior to expiration of the timer.

25. The computer-storage medium of claim 21 , the operations further comprising:

updating the storage location based on the completion of the task and responsive to the dequeue request received from the worker node.

26. The computer-storage medium of claim 21 , the operations further comprising:

sharding the in-memory task queue into a plurality of shards; and

storing the task in a first shard of the plurality of shards.

27. The computer-storage medium of claim 26 , the operations further comprising:

decoding a second enqueue request received from the data process, the second enqueue request including a second task; and

enqueuing the second task in a second shard of the plurality of shards.

28. The computer-storage medium of claim 27 , the operations further comprising:

enqueuing one or more additional tasks received after the second task, wherein the one or more additional tasks are enqueued into corresponding shards of the plurality of shards in a round-robin configuration.

29. The computer-storage medium of claim 21 , the operations further comprising:

enqueuing a second lease request received from a second worker node for a second task in the in-memory task queue; and

responsive to the enqueuing of the second lease request:

initiating a second lease of the second task to the second worker node;

initiating a lease timer for a pre-configured period, the lease timer starting a time the second lease is initiated; and

changing a state of the second task from a ready state to a leased state.

30. The computer-storage medium of claim 29 , the operations further comprising:

detecting the lease timer has expired;

performing a revocation of the second lease; and

changing the state of the second task from the leased state to the ready state.