Elastic and scalable publish/subscribe service
A system and method are disclosed for an elastic and scalable publish/subscribe scheme. Subscription information is received at a dispatcher node. A plurality of matching nodes are selected in an overlay network to store the subscription information on a computer readable storage medium. Upon receiving an event at a dispatching node, at least one of the matching nodes with the stored subscription information is selected to process the event.
1. A method for operating a publish/subscribe system, comprising:
receiving, by a dispatching node, subscription information;
selecting, by a dispatching node, a plurality of matching nodes in an overlay network to store the subscription information on a computer readable storage medium at each of the plurality of matching nodes based on each of the plurality of matching nodes being a single hop from the dispatching node;
receiving an event at a dispatching node;
selecting at least one of the matching nodes with the stored subscription information to process the event;
periodically receiving, by the dispatching node, load information from each of the matching nodes; and
adding or removing, by the dispatching node, matching nodes from the overlay network, based on the load information.
2. The method of claim 1 , wherein selecting a plurality of matching nodes in an overlay network to store the subscription information includes applying a multi-dimensional subscription space partitioning technique to partition each attribute associated with the subscription information into a plurality of subscription space segments, and assigning an entirety of the subscription information to different matching nodes which are each responsible for at least one of the subscription space segments.
3. The method of claim 2 , further comprising identifying the subscription space segments that are associated with attribute values of an incoming event using information propagated through gossiping, and identifying a set of candidate matching nodes responsible for the identified subscriptions space segments.
4. The method of claim 1 , wherein selecting at least one of the matching nodes with the stored subscription information to process the event includes selecting the matching node with a lowest predicted response time to execute a matching procedure to determine which events are to be transmitted to subscribers.
5. The method of claim 1 , further comprising periodically transmitting data associated with a matching node to a predetermined number of randomly selected matching nodes using a gossiping protocol.
6. The method of claim 5 , wherein the data indicates a liveliness value, contact information and load information associated with a matching node.
7. A method for operating a publish/subscribe system, comprising:
receiving, by a dispatching node, subscription information;
selecting, by the dispatching node, a plurality of matching nodes in an overlay network to store the subscription information on a computer readable storage medium at each of the plurality of matching nodes based on each of the plurality of matching nodes being a single hop from the dispatching node;
receiving an event at a dispatching node;
selecting the matching node with a lowest predicted response time to execute a matching procedure for determining which events are to be transmitted to subscribers;
periodically transmitting data associated with a matching node to a predetermined number of randomly selected matching nodes using a gossiping protocol;
periodically receiving, by the dispatching node, load information from each of the matching nodes; and
adding or removing, by the dispatching node, matching nodes from the overlay network, based on the load information.
8. The method of claim 7 , wherein selecting a plurality of matching nodes in an overlay network to store the subscription information includes applying a multi-dimensional subscription space partitioning technique to partition each attribute associated with the subscription information into a plurality of subscription space segments.
9. The method of claim 8 , further comprising assigning an entirety of the subscription to different matching nodes which are each responsible for at least one of the subscription space segments.
10. The method of claim 8 , further comprising identifying the subscription space segments that are associated with attribute values of an incoming event using information propagated through gossiping, and identifying a set of candidate matching nodes responsible for the identified subscriptions space segments.
11. The method of claim 7 , wherein the lowest predicted response time is computed based on at least one of: length of an event queue, message rate or match rate.