Techniques and architectures for non-blocking parallel batching

1. A method comprising:

receiving, via an interface, an event associated with a corresponding topic;

mapping, by a mapping agent, the event to a selected one of a plurality of topic partitions, the selected one of the plurality of topic partitions being associated with the corresponding topic, wherein events within each of the plurality of topic partitions are processed according to topic ordering constraint;

managing by one or more lane agents, for each topic partition of the plurality of topic partitions, a corresponding queue of a plurality of queues, each queue of the plurality of queues being communicatively coupled to and managed by a corresponding lane agent of the one or more lane agents, wherein the plurality of queues are managed independently from one another, each topic partition being associated with a corresponding one of a plurality of memory partitions of a storage system including one or more storage devices, the plurality of memory partitions being independent from one another;

placing, by the mapping agent, the event into a particular queue of the plurality of queues, the particular queue corresponding to the selected topic partition; and

coordinating a sending, by a particular lane agent of the one or more lane agents, of a batch of events from the particular queue for the selected topic partition to the corresponding memory partition, the particular queue being communicatively coupled to and managed by the particular lane agent, each lane agent being configured to coordinate sending event data from a corresponding queue of the plurality of queues to a corresponding memory partition, the particular lane agent being configured to coordinate sending event data from the particular queue.

2. The method of claim 1 , the storage devices comprising at least two different types of storage structures.

3. The method of claim 1 , the mapping comprising utilizing a hashing function.

4. The method of claim 1 , wherein coordinating sending, by the particular agent, the batch of events from the particular queue for the selected topic partition comprises:

transmitting the batch of events from the particular queue to a scheduler operatively coupled to the memory partition corresponding to the selected topic partition.

5. The method of claim 1 , further comprising:

detecting a trigger event;

wherein coordinating the sending of the batch of events is performed responsive to detecting the trigger event.

6. The method of claim 5 , wherein detecting a trigger event comprises:

determining that the particular queue has reached a particular threshold.

7. The method of claim 6 , the particular threshold being based on time.

8. The method of claim 6 , the particular threshold being based on queue size.

9. A non-transitory computer-readable medium comprising computer program instructions, the computer program instructions configurable such that, when executed by one or more processors, the computer program instructions cause the one or more processors to:

process an event received via an interface, the event being associated with a corresponding topic;

map, by a mapping agent, the event to a selected one of a plurality of topic partitions, the selected one of the plurality of topic partitions being associated with the corresponding topic, wherein events within each of the plurality of topic partitions are processed according to topic ordering constraints;

manage by one or more lane agents, for each topic partition of the plurality of topic partitions, a corresponding queue of a plurality of queues, each queue of the plurality of queues being communicatively coupled to and managed by a corresponding lane agent of the one or more lane agents, wherein the plurality of queues are managed independently from one another, each topic partition being associated with a corresponding one of a plurality of memory partitions of a storage system including one or more storage devices, the plurality of memory partitions being independent from one another;

place, by the mapping agent, the event into a particular queue of the plurality of queues, the particular queue corresponding to the selected topic partition; and

coordinate a sending, by a particular lane agent of the one or more lane agents, of a batch of events from the particular queue for the selected topic partition to the corresponding memory partition, the particular queue being communicatively coupled to and managed by the particular lane agent, each lane agent being configured to coordinate sending event data from a corresponding queue of the plurality of queues to a corresponding memory partition, the particular lane agent being configured to coordinate sending event data from the particular queue.

10. The non-transitory computer-readable medium of claim 9 , the storage devices comprising at least two different types of storage structures.

11. The non-transitory computer-readable medium of claim 9 , the mapping comprising utilizing a hashing function.

12. The non-transitory computer-readable medium of claim 9 , further comprising computer program instructions configurable such that, when executed by the one or more processors, the computer program instructions cause the one or more processors to:

coordinate sending the batch of events by transmitting the batch of events to a scheduler operatively coupled to the memory partition corresponding to the selected topic partition.

13. The non-transitory computer-readable medium of claim 9 , further comprising computer program instructions configurable such that, when executed by the one or more processors, the computer program instructions cause the one or more processors to:

detecting a trigger event;

wherein sending the batch of events is performed responsive to detecting the trigger event.

14. The non-transitory computer-readable medium of claim 13 , wherein detecting a trigger event comprises:

determining that the particular queue has reached a particular threshold.

15. A system comprising:

at least one memory system;

one or more processors coupled with the at least one memory system, the one or more processors configurable to:

process an event received via an interface, the event being associated with a corresponding topic;

map, by a mapping agent, the event to a selected one of a plurality of topic partitions, the selected one of the plurality of topic partitions being associated with the corresponding topic, wherein events within each of the plurality of topic partitions are processed according to topic ordering constraints;

manage by one or more lane agents, for each topic partition of the plurality of topic partitions, a corresponding queue of a plurality of queues, each queue of the plurality of queues being communicatively coupled to and managed by a corresponding lane agent of the one or more lane agents, wherein the plurality of queues are managed independently from one another and in parallel, each topic partition being associated with a corresponding one of a plurality of memory partitions of a storage system including one or more storage devices, the plurality of memory partitions being independent from one another;

place, by the mapping agent, the event into a particular queue of the plurality of queues, the particular queue corresponding to the selected topic partition; and

coordinate a sending, by a particular lane agent of the one or more lane agents, of a batch of events from the particular queue for the selected topic partition to the corresponding memory partition, the particular queue being communicatively coupled to and managed by the particular lane agent, each lane agent being configured to coordinate sending event data from a corresponding queue of the plurality of queues to a corresponding memory partition, the particular lane agent being configured to coordinate sending event data from the particular queue.

16. The system of claim 15 , the storage devices comprising at least two different types of storage structures.

17. The system of claim 15 , the mapping comprising utilizing a hashing function.

18. The system of claim 15 , the one or more processors further configurable to:

coordinate sending the batch of events by transmitting the batch of events to a scheduler operatively coupled to the memory partition corresponding to the selected topic partition.

19. The system of claim 15 , the one or more processors further configurable to:

detect a trigger event;

wherein coordinating the sending of the batch of events is performed responsive to detecting the trigger event.

20. The system of claim 19 , the one or more processors further configurable to:

a trigger event comprises:

detect the trigger event by determining that the particular queue has reached a particular threshold.