Dynamic queue management
Methods and systems for managing queues are disclosed. The disclosed methods involve receiving, at an interface, a request to store a message in a queue, storing the message as a row in a key-value data store, extracting at least one attribute from the message; and selecting the message for processing based on the at least one extracted attribute.
1. A method of managing queues, the method comprising:
receiving, at an interface, a request to store a message in a queue;
storing the message as a row in a key-value data store that includes at least one primary key that is composed of a first partition key and a first sort key, wherein the queue is associated with a partition in the data store and the data store includes a plurality of partitions that are each associated with a queue;
elevating the queue storing the message for analysis based on a number of messages within the queue and an existence duration of the queue;
deprioritizing at least a second queue based on a number of messages within the second queue and existence duration of the second queue;
extracting, using a processor executing instructions stored on memory, at least one attribute from the message; and
selecting, using the processor, the message for processing based on the at least one extracted attribute.
2. The method of claim 1 wherein the at least one attribute includes at least one of age of the message and sensitivity of the message.
3. The method of claim 1 wherein the partition includes a plurality of messages, and the method further includes:
performing, using the processor, an attribute analysis on each of the plurality of messages, and
elevating for processing a message within the partition based on the attribute analysis.
4. The method of claim 1 further comprising associating a unique identifier of the message to the first sort key of the queue.
5. The method of claim 1 wherein the at least one primary key is unique to the queue.
6. The method of claim 5 wherein the primary key is composed of:
an organization and a name;
an organization and a sub-division;
a country and a state; or
a country and a city.
7. A system for managing queues, the system comprising:
an interface for at least receiving a request to store a message in a queue;
a key-value data store that includes at least one primary key that is composed of a first partition key and a first sort key, wherein the queue is associated with a partition in the data store and the data store includes a plurality of partitions that are each associated with a queue; and
a processor executing instructions stored in memory configured to:
elevate the queue storing the message for analysis based on a number of messages within the queue and an existence duration of the queue,
deprioritize at least a second queue based on a number of messages within the second queue and existence duration of the second queue,
extract at least one attribute from the message, and
select the message for processing based on the at least one extracted attribute.
8. The system of claim 7 wherein the at least one attribute includes at least one of age of the message and sensitivity of the message.
9. The system of claim 7 wherein the partition includes a plurality of messages and the processor is further configured to:
perform an attribute analysis on each of the plurality of messages; and
elevate for processing a message within the partition based on the attribute analysis.
10. The system of claim 7 wherein a unique identifier of the message is associated with the first sort key of the queue.
11. The system of claim 7 wherein the processor is further configured to associate the message with a primary key that is unique to the queue.
12. The system of claim 11 wherein the primary key is composed of:
an organization and a name;
an organization and a sub-division;
a country and a state; or
a country and a city.