Distributed in-memory timer queues
Disclosed are systems and methods for implementing distributed in-memory timer queues. A plurality of queues is distributed across a plurality of nodes. The plurality of queues includes a first queue. The first queue corresponds to current timer expirations for a tenant, time bucket, and shard. A sorted set includes minute references to unacknowledged timers for a tenant and shard. A plurality of processes is created. Each of the plurality of processes is configured to be a unique consumer of the distributed plurality of queues. A first consumer group and a second consumer group are created within each of the plurality of processes. The first consumer group corresponds to the first queue. The second consumer group corresponds to the sorted set. The plurality of processes is used to proportionally consume the first queue with the first consumer group and the sorted set with the second consumer group.
1 . A method comprising:
distributing a plurality of queues across a plurality of nodes, the plurality of queues including a first queue, the first queue corresponding to timer expirations for a tenant, time bucket, and shard, wherein a sorted set includes references to unacknowledged timers for a tenant and shard;
creating a plurality of processes, each of the plurality of processes configured to be a unique consumer of the distributed plurality of queues,
creating a first consumer group and a second consumer group within each of the plurality of processes, the first consumer group corresponding to the first queue, the second consumer group corresponding to the sorted set; and
using the plurality of processes to consume the first queue with the first consumer group and the second queue with the second consumer group.
2 . The method of claim 1 , wherein an additional sorted set includes unacknowledged timers in the past for a tenant, minute, and shard.
3 . The method of claim 2 , the operations further comprising using the plurality of processes to proportionally consume the additional sorted set with the second consumer group.
4 . The method of claim 1 , the plurality of queues further including an additional queue, the additional queue corresponding to future timers for a tenant, time bucket, and shard.
5 . The method of claim 1 , wherein the consuming of the timers from the additional queue includes placing them on an in-memory minheap.
6 . The method of claim 1 , the operations further comprising removing the sorted set when all unacknowledged timers have been processed.
7 . The method of claim 1 , wherein the consuming of the first queue and the sorted set includes firing the timer to an endpoint destination.
8 . A system comprising:
one or more computer processors; and
one or more memories;
a set of instructions stored in the one or more computer memories, the sect of instructions causing the one or more computer processors to perform operations, the operations comprising:
distributing a plurality of queues across a plurality of nodes, the plurality of queues including a first queue, the first queue corresponding to timer expirations for a tenant, time bucket, and shard, wherein a sorted set includes references to unacknowledged timers for a tenant and shard;
creating a plurality of processes, each of the plurality of processes configured to be a unique consumer of the distributed plurality of queues;
creating a first consumer group and a second consumer group within each of the plurality of processes, the first consumer group corresponding to the first queue, the second consumer group corresponding to the sorted set; and
using the plurality of processes to proportionally consume the first queue with the first consumer group and the second queue with the second consumer group.
9 . The system of claim 8 , wherein an additional sorted set includes unacknowledged timers in the past for a tenant, time bucket, and shard.
10 . The system of claim 9 , the operations further comprising using the plurality of processes to proportionally consume the additional sorted set with the second consumer group.
11 . The system of claim 8 , the plurality of queues further including an additional queue, the additional queue corresponding to future timers for a tenant, time bucket, and shard.
12 . The system of claim 8 , wherein the consuming of the timers from the sorted set includes placing them on an in-memory minheap.
13 . The system of claim 8 , the operations further comprising removing the sorted set when all unacknowledged timers have been processed.
14 . The system of claim 8 , wherein the consuming of the first queue and the sorted set includes firing the timer to an endpoint destination.
15 . A non-transitory computer-readable storage medium storing a set of instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform operations, the operations comprising:
distributing a plurality of queues across a plurality of nodes, the plurality of queues including a first queue, the first queue corresponding to current timer expirations for a tenant, time bucket, and shard, wherein a sorted set includes minute references to unacknowledged timers for a tenant and shard;
creating a plurality of processes, each of the plurality of processes configured to be a unique consumer of the distributed plurality of queues;
creating a first consumer group and a second consumer group within each of the plurality of processes, the first consumer group corresponding to the first queue, the second consumer group corresponding to the sorted set; and
using the plurality of processes to proportionally consume the first queue with the first consumer group and the second queue with the second consumer group.
16 . The non-transitory computer-readable storage medium of claim 15 , wherein an additional sorted set includes unacknowledged timers in the past for a tenant, time bucket, and shard.
17 . The non-transitory computer-readable storage medium of claim 16 , the operations further comprising using the plurality of processes to proportionally consume the additional sorted set with the second consumer group.
18 . The non-transitory computer-readable storage medium of claim 15 , the plurality of queues further including an additional queue, the additional queue corresponding to future timers for a tenant, time bucket, and shard.
19 . The non-transitory computer-readable storage medium of claim 15 , wherein the consuming of the timers from the sorted set includes placing them on an in-memory minheap.
20 . The non-transitory computer-readable storage medium of claim 15 , the operations further comprising removing the sorted set when all unacknowledged timers have been processed.