Processing high volume network data
Disclosed are a system comprising a computer-readable storage medium storing at least one program, and a computer-implemented method for event messaging over a network. A subscription interface receives data indicative of a subscription request for sessionized data. An allocation module allocates a sessionizer bank linked to the subscription request. A messaging interface module provisions identifiers linked to the respective processing engines of the sessionizer bank. The messaging interface module registers the allocated sessionizer bank as available to process event messages matching the subscription request by providing the provisioned identifiers. The messaging interface module receives event messages from a producer device linked by a collection server to a selected one of the processing engines of the sessionizer bank. The selected one of the processing engine processes the received event messages in accordance with session rule data linked to the subscription request to generate sessionized data.
1 . A system for processing event messages comprising, the system comprising:
one or more processors; and
a non-transitory memory to store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving a first message comprising advertisement data indicating an availability of a first machine to receive data messages linked to a topic;
computing at least a first value based on the topic and an identifier of the first machine;
associating the first value with the first machine;
receiving an event message linked to the topic from a second machine, the event message comprising a second value; and
providing the event message to the first machine based at least on a correspondence between the first value and the second value, wherein the correspondence is determined by traversing a perimeter of a logical ring in a clockwise or anti-clockwise direction starting at a point of a hash of the second value until reaching a first hash value on the logical ring corresponding to the first machine based at least on correlating the hash on the second value to the first hash value, wherein the logical ring is formed for the topic by assigning one or more hash values to consumer devices using consistent hashing to partition a consumer cluster, wherein the logical ring comprises a ring of hash values instantiated for the topic.
2 . The system of claim 1 , the one or more processors to perform operations further comprising:
storing the first value to a registry that associates the first value to the first machine.
3 . The system of claim 1 , the one or more processors to perform operations further comprising:
determining the correspondence between the first value and the second value based at least on matching a hash of the second value to the first value.
4 . The system of claim 1 , the one or more processors to perform operations further comprising:
computing the first value to link the first machine and the second machine based on a second message that identifies the second machine as available to provide the data messages linked to the topic.
5 . The system of claim 4 , wherein the second message comprises a broadcast discovery message that includes at least one of a topic identifier for the topic, and an identifier for the second machine.
6 . The system of claim 1 , the one or more processors to perform operations further comprising:
computing the first value based on a consistent hashing ring associated with the topic.
7 . A method for processing event messages, the method comprising:
receiving first data identifying one or more consumer devices available to receive data messages linked to a topic;
linking a plurality of values to respective consumer devices of the one or more consumer devices based at least on the first data;
accessing from a producer device a first event message linked to the topic, the first event message comprising a key value; and
providing the first event message to a selected one of the one or more consumer devices based on a comparison performed using the key value and the plurality of values of the respective consumer devices, wherein the comparison is performed by traversing a perimeter of a logical ring in a clockwise or anti-clockwise direction starting at a point of a hash of the key value until reaching a first hash value on the logical ring corresponding to the selected one of the one or more consumer devices based at least on correlating the hash of the key value to the first hash value, wherein the logical ring is formed for the topic by assigning one or more hash values to consumer devices using consistent hashing to partition a consumer cluster, wherein the logical ring comprises a ring of hash values instantiated for the topic.
8 . The method of claim 7 , the method further comprising:
receiving second data identifying the producer device as available to provide the data messages linked to the topic.
9 . The method of claim 8 , wherein the second data comprises a broadcast discovery message that includes at least one of a topic identifier for the topic, and an identifier for the producer device.
10 . The method of claim 7 , the method further comprising:
computing the plurality of values based on a consistent hashing ring associated with the topic.
11 . The method of claim 7 , the method further comprising:
performing the comparison using the key value and the plurality of values based at least on matching a hash of the key value to the plurality of values.
12 . The method of claim 7 , the method further comprising:
computing for at least a first consumer device of the one or more consumer devices, at least a first value based on the topic and an identifier of the first consumer device.
13 . A system for processing event messages comprising, the system comprising:
one or more processors; and
a non-transitory memory to store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving first data comprising advertisement data identifying one or more consumer devices available to receive data messages linked to a topic;
computing, for each of the one or more consumer devices, at least a first value based on the topic and identifiers for the one or more consumer devices;
linking each respective consumer device of the one or more consumer devices to a respective first value;
accessing from a producer device a first event message linked to the topic, the first event message comprising a key value; and
providing the first event message to a selected one of the one or more consumer devices based on a comparison performed using the key value and the first value for each of the respective one or more consumer devices, wherein the comparison is performed by traversing a perimeter of a logical ring in a clockwise or anti-clockwise direction starting at a point of a hash of the key value until reaching a first hash value on the logical ring corresponding to the selected one of the one or more consumer devices based at least on correlating the hash of the key value to the first hash value, wherein the logical ring is formed for the topic by assigning one or more hash values to consumer devices using consistent hashing to partition a consumer cluster, wherein the logical ring comprises a ring of hash values instantiated for the topic.
14 . The system of claim 13 , the one or more processors to perform operations further comprising:
receiving second data identifying the producer device as available to provide the data messages linked to the topic.
15 . The system of claim 14 , wherein the second data comprises a broadcast discovery message that includes at least one of a topic identifier for the topic, and an identifier for the producer device.
16 . The system of claim 13 , the one or more processors to perform operations further comprising:
computing the first value based on a consistent hashing ring associated with the topic.
17 . The system of claim 13 , the one or more processors to perform operations further comprising:
performing the comparison using the key value and the first value for each of the respective one or more consumer devices based at least on matching a hash of the key value to the first value for each of the respective one or more consumer devices.