Intelligent event streaming
Various embodiments include devices and methods structured to provide an intelligent event processing offload, which can facilitate an efficient, geo-distributed tiered scalable event streaming and processing interconnect through an embedded adapter interconnect. Event streams received in a data handler from one or more networks can be separated into partitions. The loading of the event streams into the partitions can be tracked to balance the loading by controlling the number of partitions used. Packets of each partition can be sent to a respective reactive router that resides in a controlling application for further processing. Additional devices, systems, and methods can be implemented in a variety of applications.
1. A device operable to handle event streams for processing, comprising:
a memory storage comprising instructions; and
one or more processors in communication with the memory storage, wherein the one or more processors execute the instructions to perform operations comprising:
receiving event streams in a data handler, each event stream originating from a respective source network;
separating the event streams from the data handler into a number of partitions based on topics of the event streams;
tracking loading of the event streams into the number of partitions to balance the loading by controlling the number of partitions, wherein the tracking includes tracking input throughput peaks and output throughput peaks of the number of partitions;
resizing the number of partitions based on captured peak values and differences with respect to a maximum throughput using heuristics and close loop control; and
sending packets of each partition, associated with the event streams, to a respective reactive router that resides in a controlling application.
2. The device of claim 1 , wherein the differences with respect to the maximum throughput are differences between a maximum ceiling throughput and measured input throughput.
3. The device of claim 1 , wherein the operations include
examining, by the respective reactive router, a payload of each packet and determining a module of the controlling application to process the packet based on the examination; and
inserting, by the respective reactive router, each packet into an input queue of the determined module.
4. The device of claim 3 , wherein the operations include creating one or more additional queues, by the respective reactive router, when one or more input queues of modules of the controlling application become unbalanced.
5. The device of claim 3 , wherein the operations include re-directing packets, by the respective reactive router, using a payload filter policy that includes a quality of service parameter and peer information.
6. The device of claim 3 , wherein the operations include the respective reactive router selectively distributing individual packets to software defined network (SDN) and/or Internet of things (IoT) network nodes using a cache that maintains registration of the SDN and/or IoT network nodes based on domains.
7. A computer-implemented method comprising:
receiving event streams in a data handler, each event stream originating from a respective source network;
separating, using a processor, the event streams from the data handler into a number of partitions based on topics of the event streams;
tracking loading of the event streams into the number of partitions to balance the loading by controlling the number of partitions, wherein the tracking includes tracking input throughput peaks and output throughput peaks of the number of partitions;
resizing the number of partitions based on captured peak values and differences with respect to a maximum throughput using heuristics and close loop control; and
sending packets of each partition, associated with the event streams, to a respective reactive router that resides in a controlling application.
8. The computer-implemented method of claim 7 , wherein the differences with respect to the maximum throughput are differences between a maximum ceiling throughput and measured input throughput.
9. The computer-implemented method of claim 7 , wherein the method includes
examining, by the respective reactive router, a payload of each packet and determining a module of the controlling application to process the packet based on the examination; and
inserting, by the respective reactive router, each packet into an input queue of the determined module.
10. The computer-implemented method of claim 9 , wherein the method includes creating one or more additional queues, by the respective reactive router, when one or more input queues of modules of the controlling application become unbalanced.
11. The computer-implemented method of claim 9 , wherein the method includes re-directing packets, by the respective reactive router, using a payload filter policy that includes a quality of service parameter and peer information.
12. The computer-implemented method of claim 9 , wherein method includes the respective reactive router selectively distributing individual packets to software defined network (SDN) and/or Internet of things (IoT) network nodes using a cache that maintains registration of the SDN and/or IoT network nodes based on domains.
13. A non-transitory computer-readable medium storing computer instructions, that when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving event streams in a data handler, each event stream originating from a respective source network;
separating the event streams from the data handler into a number of partitions based on topics of the event streams;
tracking loading of the event streams into the number of partitions to balance the loading by controlling the number of partitions, wherein the tracking includes tracking input throughput peaks and output throughput peaks of the number of partitions;
resizing the number of partitions based on captured peak values and differences with respect to a maximum throughput using heuristics and close loop control; and
sending packets of each partition, associated with the event streams, to a respective reactive router that resides in a controlling application.
14. The non-transitory computer-readable medium of claim 13 , wherein the operations include
examining, by the respective reactive router, a payload of each packet and determining a module of the controlling application to process the packet based on the examination; and
inserting, by the respective reactive router, each packet into an input queue of the determined module.
15. The non-transitory computer-readable medium of claim 14 , wherein the operations include creating one or more additional queues, by the respective reactive router, when one or more input queues of modules of the controlling application become unbalanced.
16. The non-transitory computer-readable medium of claim 14 , wherein the operations include re-directing packets, by the respective reactive router, using a payload filter policy that includes a quality of service parameter and peer information.
17. The non-transitory computer-readable medium of claim 14 , wherein the operations include the respective reactive router selectively distributing individual packets to software defined network (SDN) and/or Internet of things (IoT) network nodes using a cache that maintains registration of the SDN and/or IoT network nodes based on domains.