Efficient message queuing service using multiplexing
Embodiments of the present invention are directed to facilitating efficient message queueing. In particular, embodiments herein describe, among other things, a redelivery monitor used to monitor when to redeliver messages, or tasks, for reprocessing based on expiration of a redelivery deadline. In this regard, markers indicating processing states for tasks being processed are read by the redelivery monitor. When the processing state indicates that processing is ongoing, the redelivery deadline is extended such that a message or task is not redelivered for processing while the message or task is being processed.
1. A computer-implemented method for performing data processing in a streaming manner, the computer-implemented method comprising:
receiving a first set of tasks at a first logical queue of a plurality of logical queues, each task of the first set of tasks including an identifier indicating a first data collection request associated with the first set of tasks;
receiving a second set of tasks at a second logical queue of the plurality of logical queues, each task of the second set of tasks including an identifier indicating a second data collection request associated with the second set of tasks; and
obtaining, at a queue topic containing tasks to be processed, the first set of tasks associated with the first data collection request from the first logical queue and the second set of tasks associated with the second data collection request from the second logical queue, wherein each of the tasks of the first set of tasks and the second set of tasks are distributed to any partition of a plurality of partitions of the queue topic.
2. The method of claim 1 , wherein each partition of the plurality of partitions holds a set of tasks associated with multiple data collection requests, wherein each of the tasks of the first set of tasks and the second set of tasks are assigned to one of the partitions in accordance with a random, uniform distribution.
3. The method of claim 1 , wherein each of the tasks of the first set of tasks and the second set of tasks are distributed to any partition in a random manner.
4. The method of claim 1 , wherein tasks contained within various partitions of the plurality of partitions are read in parallel.
5. The method of claim 1 , wherein the plurality of partitions of the queue topic enable tasks within the plurality of partitions to be read from the queue topic in parallel.
6. The method of claim 1 , wherein the first logical queue receives the first set of tasks associated with the first data collection request from a producer.
7. The method of claim 1 , wherein all tasks for all data collection requests are obtained at the queue topic via the plurality of logical queues.
8. The method of claim 1 , wherein tasks of the first set of tasks are distributed across partitions of the plurality of partitions of the queue topic and tasks of the second set of tasks are distributed across partitions of the plurality of partitions.
9. The method of claim 1 , wherein a first partition of the plurality of partitions includes a task from the first set of tasks and a task from the second set of tasks.
10. The method of claim 1 , wherein a consumer reads tasks from each partition of the plurality of partitions and filters the read tasks to process the read tasks that correspond to a data collection request of interest.
11. The method of claim 1 , wherein a consumer reads tasks from each partition of the plurality of partitions and filters the read tasks to process the read tasks that correspond to a data collection request of interest and drop the read tasks that do not correspond to the data collection request of interest.
12. The method of claim 1 further comprising multiplexing a set of logical queues to the queue topic to communicate messages associated with multiple data collection requests to the queue topic.
13. The method of claim 1 further comprising:
reading, via a consumer, a task from the queue topic;
initiating processing of the read task;
monitoring the task processing; and
writing markers indicating a processing state of the task to a markers topic.
14. The method of claim 1 , wherein the first set of tasks and the second set of tasks are consumed via a message queuing service that performs data processing in the streaming manner, the message queuing service including the first logical queue, the second logical queue, and the queue topic.
15. A computing system comprising:
a processor; and
computer storage memory having computer-executable instructions stored thereon which, when executed by the processor, configure the computing system to:
receive a first set of tasks at a first logical queue of a plurality of logical queues, each task of the first set of tasks including an identifier indicating a first data collection request associated with the first set of tasks;
receive a second set of tasks at a second logical queue of the plurality of logical queues, each task of the second set of tasks including an identifier indicating a second data collection request associated with the second set of tasks; and
obtain, at a queue topic containing tasks to be processed, the first set of tasks associated with the first data collection request from the first logical queue and the second set of tasks associated with the second data collection request from the second logical queue, wherein each of the tasks of the first set of tasks and the second set of tasks are distributed to any one partition of a plurality of partitions of the queue topic.
16. One or more computer storage media having computer-executable instructions embodied thereon that, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising:
receiving a first set of tasks at a first logical queue of a plurality of logical queues, each task of the first set of tasks including an identifier indicating a first data collection request associated with the first set of tasks;
receiving a second set of tasks at a second logical queue of the plurality of logical queues, each task of the second set of tasks including an identifier indicating a second data collection request associated with the second set of tasks; and
obtaining, at a queue topic containing tasks to be processed, the first set of tasks associated with the first data collection request from the first logical queue and the second set of tasks associated with the second data collection request from the second logical queue, wherein each of the tasks of the first set of tasks and the second set of tasks are distributed to any one partition of a plurality of partitions of the queue topic.
17. The one or more computer storage media of claim 16 , wherein all tasks for all data collection requests are obtained at the queue topic via the plurality of logical queues.
18. The one or more computer storage media of claim 16 , wherein tasks of the first set of tasks are distributed across partitions of the plurality of partitions of the queue topic and tasks of the second set of tasks are distributed across partitions of the plurality of partitions.
19. The one or more computer storage media of claim 16 , wherein a first partition of the plurality of partitions includes a task from the first set of tasks and a task from the second set of tasks.
20. The one or more computer storage media of claim 16 , wherein a consumer reads tasks from each partition of the plurality of partitions and filters the read tasks to process the read tasks that correspond to a data collection request of interest.