Mapping logical partitions for event data to a plurality of queues
According to examples, an apparatus may include a processor and a memory on which are stored machine-readable instructions that, when executed by the processor, may cause the processor to receive event data to be placed in a plurality of queues. The processor may determine logical partitions for the received event data and may store the event data in the plurality of queues based on the determined logical partitions. A first queue of the plurality of queues may be mapped to a first logical partition range and a second queue of the plurality of queues may be mapped to a second logical partition range. Based on respective loads at the plurality of queues, the processor may update a mapping of the determined logical partitions to the plurality of queues by transferring logical partitions between the first queue and the second queue.
1. An apparatus comprising:
a processor; and
a memory on which is stored machine-readable instructions that when executed by the processor, cause the processor to:
receive event data to be placed in a plurality of queues;
determine logical partitions for the received event data;
store the event data in the plurality of queues based on the determined logical partitions, a first logical partition range being mapped to a first queue of the plurality of queues and a second logical partition range being mapped to a second queue of the plurality of queues;
determine respective loads at the plurality of queues in real-time; and
based on the respective loads at the plurality of queues, update a mapping of the determined logical partitions to the plurality of queues, wherein:
a respective determined logical partition is mapped to a respective queue of the plurality of queues; and
updating the mapping of the determined logical partitions to the plurality of queues comprises updating the mapping of at least one of the determined logical partitions from the first queue to the second queue to increase a balance of the respective loads across the plurality of queues based on a calculated probability of a respective determined logical partition to be transferred to another queue of the plurality of queues.
2. The apparatus of claim 1 , wherein the first logical partition range is correlated to a first number of the determined logical partitions assigned to the first queue, and the second logical partition range is correlated to a second number of the determined logical partitions assigned to the second queue, the first number of the determined logical partitions being different than the second number of the determined logical partitions.
3. The apparatus of claim 1 , wherein the instructions further cause the processor to:
identify partition keys in respective fields of the received event data; and
determine the logical partitions for the received event data based on the identified partition keys.
4. The apparatus of claim 3 , wherein a number of the determined logical partitions is greater than a number of the plurality of queues, and the number of the determined logical partitions is proportional to a number of the identified partition keys.
5. The apparatus of claim 1 , wherein the instructions further cause the processor to:
initially set the first logical partition range and the second logical partition range to comprise the same number of logical partitions with respect to each other during an initialization stage to evenly distribute the mapping of the determined logical partitions across the plurality of queues.
6. The apparatus of claim 1 , wherein the instructions further cause the processor to:
iteratively update the mapping of the determined logical partitions to the plurality of queues until a convergence is reached in which the respective loads at the plurality of queues are balanced.
7. The apparatus of claim 1 , wherein the instructions further cause the processor to:
determine a load at the first queue;
based on a determination that the load at the first queue is greater than a predefined threshold value, update the mapping of the determined logical partitions for the first queue to decrease a number of the determined logical partitions mapped to the first queue; and
based on a determination that the load at the first queue is less than a second predefined threshold value, update the mapping of the determined logical partitions for the first queue to increase the number of the determined logical partitions mapped to the first queue.
8. The apparatus of claim 7 , wherein the instructions further cause the processor to:
determine the load at the first queue based on a metric correlated to the first queue, the metric including a number of events written per second, a data size for events written per second, a rate of reading data from the first queue, or a combination thereof.
9. The apparatus of claim 7 , wherein the instructions further cause the processor to:
based on a determination that the load at the first queue is greater than the predefined threshold value, transfer the mapping of a certain number of the determined logical partitions from the first queue to the second queue of the plurality of queues that has a load that is less than the second predefined threshold value.
10. A method comprising:
receiving, by a processor, a plurality of event data to be placed in a plurality of queues;
identifying, by the processor, partition keys associated with the received event data;
determining, by the processor and for each of the event data, a logical partition for the event data based on the identified partition key associated with the event data;
storing, by the processor, the received event data in the plurality of queues based on the determined logical partitions for the event data, each of the logical partition ranges being mapped to one of the plurality of queues; and
based on loads at the plurality of queues, updating, by the processor, a mapping of at least one of the logical partition ranges to at least one of the plurality of queues to increase a balance of the loads across the plurality of queues based on a calculated probability of a respective determined logical partition to be transferred to another queue within the plurality of queues.
11. The method of claim 10 , wherein a number of the determined logical partitions is greater than a number of the plurality of queues, and the number of the determined logical partitions is proportional to a number of the identified partition keys.
12. The method of claim 10 , further comprising:
initially setting the logical partition ranges to comprise the same number of logical partitions with respect to each other during an initialization stage to evenly distributing the mapping of the determined logical partitions across the plurality of queues.
13. The method of claim 10 , further comprising:
iteratively updating the mapping of the determined logical partitions to the plurality of queues until a convergence is reached in which the respective loads at the plurality of queues are balanced.
14. The method of claim 10 , further comprising:
determining a load at a first queue among the plurality of queues;
determining whether the load at the first queue is greater than a predefined threshold value;
based on a determination that the load at the first queue is greater than the predefined threshold value, decreasing the number of the determined logical partitions mapped to the first queue; and
based on a determination that the load at the first queue is less than a second predefined threshold value, increasing the number of the determined logical partitions mapped to the first queue.
15. The method of claim 14 , further comprising:
determining the load at the first queue based on a metric correlated to the first queue, the metric including a number of events written per second, a data size for events written per second, a rate of reading data from the first queue, or a combination thereof.
16. The method of claim 14 , further comprising:
based on a determination that the load at the first queue is greater than the predefined threshold value, transferring the mapping of a certain number of the determined logical partitions from the first queue to a second queue among the plurality of queues that has a load that is less than the second predefined threshold value.
17. A non-transitory computer-readable medium on which is stored computer-readable instructions that, when executed by a processor, cause the processor to:
receive event data to be placed in a plurality of queues;
determine logical partitions for the received event data;
map a first range of the determined logical partitions to a first queue among the plurality of queues;
store the received event data having logical partitions that are within the first range in the first queue;
determine a load at the first queue; and
based on a determination that the load at the first queue is greater than a predefined threshold value, update a mapping of the determined logical partitions to the first queue from the first range to a second range of the determined logical partitions based on a calculated probability that at least a subset of the determined logical partitions within the first range is to be transferred to another queue of the plurality of queues.
18. The non-transitory computer-readable medium of claim 17 , wherein the instructions cause the processor to:
identify partition keys in respective fields of the received event data; and
determine the logical partitions for the received event data based on the identified partition keys,
wherein a number of the determined logical partitions is greater than a number of the plurality of queues, and the number of the determined logical partitions is proportional to a number of the identified partition keys.
19. The non-transitory computer-readable medium of claim 17 , wherein the instructions cause the processor to:
iteratively update the mapping of logical partition ranges associated with the determined logical partitions to the first queue.
20. The non-transitory computer-readable medium of claim 17 , wherein the instructions cause the processor to:
determine a second queue among the plurality of queues that has a load that is less than a second predefined threshold value that is lower than the predefined threshold value; and
based on a determination that the load at the first queue is greater than the predefined threshold value, transfer the mapping of at least one of the determined logical partitions from the first queue to the second queue.