Using keep-alive markers to extend redelivery deadlines
Embodiments of the present invention are directed to facilitating efficient message queueing. In particular, embodiments herein describe, among other things, a redelivery monitor used to monitor when to redeliver messages, or tasks, for reprocessing based on expiration of a redelivery deadline. In this regard, markers indicating processing states for tasks being processed are read by the redelivery monitor. When the processing state indicates that processing is ongoing, the redelivery deadline is extended such that a message or task is not redelivered for processing while the message or task is being processed.
1. A computer-implemented method comprising:
reading a task in a queue topic;
initiating processing of the task;
monitoring the processing of the task to determine that the task is being processed; and
based on the task being processed, writing a keep-alive marker to a markers topic, wherein the keep-alive marker provides an indication to extend a redelivery deadline for redelivering the task to the queue topic for reprocessing of the task.
2. The computer-implemented method of claim 1 , wherein the task is provided, via a discover coordinator, to a partition of the queue topic.
3. The computer-implemented method of claim 1 further comprising writing a start marker to a partition of the markers topic.
4. The computer-implemented method of claim 1 further comprising committing an offset upon reading the task in the queue topic.
5. The computer-implemented method of claim 1 , wherein the keep-alive marker is written to a same partition of the markers topic as a start marker.
6. The computer-implemented method of claim 1 , wherein the processing of the task is initiated via a connector.
7. The computer-implemented method of claim 1 , wherein the keep-alive marker is written to the markers topic on a periodic basis.
8. The computer-implemented method of claim 1 further comprising determining that the processing of the task is completed and, based on the completion, writing an end marker to the markers topic.
9. The computer-implemented method of claim 1 further comprising, monitoring, via a redelivery monitor, a state of the task processing by reading, from the markers topic, a set of markers associated with the task.
10. The computer-implemented method of claim 1 further comprising:
reading, from the markers topic, the keep-alive marker; and
based on the keep-alive marker read from the markers topic, extending the redelivery deadline for redelivering the task to the queue topic for reprocessing of the task.
11. The computer-implemented method of claim 1 further comprising:
determining, at a redelivery monitor, that the redelivery deadline or the extended redelivery deadline has expired; and
based on the expiration, redelivering the task to the queue topic for reprocessing of the task.
12. The computer-implemented method of claim 1 , wherein extending the redelivery deadline comprises:
determining a refresh time by which to extend the redelivery deadline, the refresh time comprising an amount of time or a time; and
determining the extended redelivery deadline using the redelivery deadline and the refresh time.
13. The computer-implemented method of claim 1 , wherein the redelivery deadline comprises a timestamp associated with a first marker plus a refresh time comprising a duration of time, the first marker comprising the keep-alive marker or a start marker.
14. A computing system comprising:
a processor; and
computer storage memory having computer-executable instructions stored thereon which, when executed by the processor, configure the computing system to:
read a task in a queue topic;
initiate processing of the task;
monitor the processing of the task to determine that the task is being processed; and
based on the task being processed, write a keep-alive marker to a markers topic, wherein the keep-alive marker provides an indication to extend a redelivery deadline for redelivering the task to the queue topic for reprocessing of the task.
15. One or more computer storage media having computer-executable instructions embodied thereon that, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising:
reading a task in a queue topic;
initiating processing of the task;
monitoring the processing of the task to determine that the task is being processed; and
based on the task being processed, writing a keep-alive marker to a markers topic, wherein the keep-alive marker provides an indication to extend a redelivery deadline for redelivering the task to the queue topic for reprocessing of the task.
16. The media of claim 15 , wherein the task is provided, via a discover coordinator, to a partition of the queue topic.
17. The media of claim 15 further comprising writing a start marker to a partition of the markers topic.
18. The media of claim 15 further comprising committing an offset upon reading the task in the queue topic.
19. The media of claim 15 , wherein the keep-alive marker is written to a same partition of the markers topic as a start marker.
20. The media of claim 15 , wherein extending the redelivery deadline comprises:
determining a refresh time by which to extend the redelivery deadline, the refresh time comprising an amount of time or a time; and
determining the extended redelivery deadline using the redelivery deadline and the refresh time.