Channel management in scalable messaging system
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for assigning channels in a messaging system. An example method includes: providing a plurality of channels, wherein each channel includes an ordered plurality of messages; assigning each channel to a channel bucket to form a plurality of channel buckets, wherein each channel bucket includes at least two channels from the plurality of channels; and assigning each channel bucket to a node selected from a plurality of nodes, wherein the node includes one or more buffers for storing messages according to the order from the channel buckets of the node.
1. A computer-implemented method, comprising:
providing a plurality of channels, wherein each channel comprises an ordered plurality of messages;
assigning, by one or more computer processors, each channel to a channel bucket to form a plurality of channel buckets, wherein each channel bucket comprises at least two channels from the plurality of channels;
distributing a listing of assignments of channels to channel buckets to a plurality of nodes processing the channels; and
assigning, by the one or more computer processors, each channel bucket to a node selected from the plurality of nodes, wherein the node comprises one or more buffers for storing messages according to the order from the channel buckets of the node.
2. The method of claim 1 , wherein assigning each channel to the channel bucket to form the plurality of channel buckets comprises:
assigning channels to channel buckets according to software applications using the channels.
3. The method of claim 2 , wherein each channel in the channel bucket comprises messages associated with a unique software application.
4. The method of claim 1 , wherein assigning each channel to the channel bucket to form the plurality of channel buckets comprises:
assigning a high-throughput channel to an isolated channel bucket,
wherein the isolated channel bucket includes no other channels.
5. The method of claim 1 , wherein the node for the channel bucket comprises a chain of peer nodes, and wherein a copy of the messages for the channel bucket is stored in each peer node.
6. The method of claim 1 , comprising:
migrating a first channel bucket from a first node to a second node within the plurality of nodes.
7. The method of claim 6 , wherein the first channel bucket is migrated from the first node to the second node to adjust a workload distribution among the plurality of nodes.
8. The method of claim 1 , comprising:
monitoring a distribution of workloads of nodes among the plurality of nodes.
9. The method of claim 8 , wherein the node is selected for each channel bucket based on the distribution of workloads among the plurality of nodes.
10. The method of claim 1 , comprising:
storing mapping information locally on each node of the plurality of nodes,
wherein the mapping information comprises a mapping between channel buckets and nodes.
11. A system, comprising:
one or more computer processors programmed to perform operations to:
provide a plurality of channels, wherein each channel comprises an ordered plurality of messages;
assign each channel to a channel bucket to form a plurality of channel buckets, wherein each channel bucket comprises at least two channels from the plurality of channels;
distribute a listing of assignments of channels to channel buckets to a plurality of nodes processing the channels; and
assign each channel bucket to a node selected from the plurality of nodes, wherein the node comprises one or more buffers for storing messages according to the order from the channel buckets of the node.
12. The system of claim 11 , wherein to assign each channel to the channel bucket to form the plurality of channel buckets the one or more computer processors are further to:
assign channels to channel buckets according to software applications using the channels.
13. The system of claim 12 , wherein each channel in the channel bucket comprises messages associated with a unique software application.
14. The system of claim 11 , wherein to assign each channel to the channel bucket to form the plurality of channel buckets the one or more computer processors are further to:
assign a high-throughput channel to an isolated channel bucket,
wherein the isolated channel bucket includes no other channels.
15. The system of claim 11 , wherein the node for the channel bucket comprises a chain of peer nodes, and wherein a copy of the messages for the channel bucket is stored in each peer node.
16. The system of claim 11 , wherein the operations further to:
migrate a first channel bucket from a first node to a second node within the plurality of nodes.
17. The system of claim 11 , wherein the operations further to:
monitor a distribution of workloads of nodes among the plurality of nodes.
18. The system of claim 17 , wherein the node is selected for each channel bucket based on the distribution of workloads among the plurality of nodes.
19. The system of claim 11 , wherein the operations further to:
store mapping information locally on each node of the plurality of nodes,
wherein the mapping information comprises a mapping between channel buckets and nodes.
20. A non-transitory computer-readable medium having instructions stored thereon that when executed by one or more computer processors cause the one or more computer processors to:
provide a plurality of channels, wherein each channel comprises an ordered plurality of messages;
assign, by the one or more computer processors, each channel to a channel bucket to form a plurality of channel buckets, wherein each channel bucket comprises at least two channels from the plurality of channels;
distribute a listing of assignments of channels to channel buckets to a plurality of nodes processing the channels and
assign, by the one or more computer processors, each channel bucket to a node selected from the plurality of nodes, wherein the node comprises one or more buffers for storing messages according to the order from the channel buckets of the node.