REPARTITIONING A TOPIC IN A PUBLISH-SUBSCRIBE MESSAGE SYSTEM
A system, method, and apparatus are provided for repartitioning a topic of a publish-subscribe message system. The topic is originally configured with N partitions (N>1) hosted by multiple brokers for storing messages to be consumed by multiple consumers. The repartitioning process causes one or more collections of partitions to be created in addition to the original collection of N partitions. Afterward, when a new message is received for the topic and has an associated partition key, the key is processed (e.g., hashed) once to identify a target collection of partitions and is then reprocessed to identify a destination partition within the target collection. Consumers may be automatically subscribed to new partitions. For example, a consumer subscribed to the i th partition of the original N partitions may be subscribed to the i th partition of each additional collection of partitions.
1 . A method comprising:
operating a publish-subscribe message system that includes a message topic having N original partitions (N>1);
during operation of the publish-subscribe message system, reconfiguring the topic to include multiple collections of partitions, wherein one of the multiple collections comprises the N original partitions;
obtaining a first message for the topic, wherein the first message has an associated partition key;
processing the partition key to identify a target collection of partitions of the topic; and
processing the partition key to identify a destination partition in the target collection of partitions; and
storing the first message in the destination partition of the target collection of partitions.
2 . The method of claim 1 , further comprising, at a first consumer of the publish-subscribe message system:
after said configuring, but before said reconfiguring, subscribing to a first partition of the N original partitions; and
after said reconfiguring, subscribing to the first partition of each of the multiple collections of partitions.
3 . The method of claim 1 , wherein said repartitioning comprises:
creating one or more additional collections of N partitions; and
for each consumer subscribed to an i th partition in the original N partitions, subscribing the consumer to the i th partition in each of the one or more additional collections of N partitions.
4 . The method of claim 1 , wherein said repartitioning comprises:
adding, to each partition of the N original partitions, a message directing a consumer of the partition to subscribe to one or more specified new partitions.
5 . The method of claim 1 , wherein said repartitioning comprises:
adding, to each partition of the N original partitions, a message informing a consumer of the partition that it has been subscribed to one or more specified new partitions.
6 . The method of claim 1 , wherein consumers of the topic share no state data.
7 . The method of claim 1 , wherein:
said obtaining, said processing, and said processing are performed by a publisher within the publish-subscribe message system.
8 . An apparatus, comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the apparatus to:
during operation of a publish-subscribe message system that includes a message topic having N original partitions (N>1), receive notification regarding reconfiguration of the topic to include multiple collections of partitions, wherein one of the multiple collections comprises the N original partitions;
obtain a first message for the topic, wherein the first message has an associated partition key;
process the partition key to identify a target collection of partitions of the topic;
process the partition key to identify a destination partition in the target collection of partitions; and
forward the first message for storage in the destination partition of the target collection of partitions.
9 . The apparatus of claim 8 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:
after said configuring, but before said reconfiguring, cause a first consumer of the publish-subscribe message system to subscribe to a first partition of the N original partitions; and
after said reconfiguring, cause the first consumer to subscribe to the first partition of each of the multiple collections of partitions.
10 . The apparatus of claim 8 , wherein said repartitioning comprises:
creation of one or more additional collections of N partitions; and
for each consumer subscribed to an i th partition in the original N partitions, subscription of the consumer to the i th partition in each of the one or more additional collections of N partitions.
11 . The apparatus of claim 8 , wherein said repartitioning comprises:
an addition, to each partition of the N original partitions, of a message directing a consumer of the partition to subscribe to one or more specified new partitions.
12 . The apparatus of claim 8 , wherein said repartitioning comprises:
an addition, to each partition of the N original partitions, of a message informing a consumer of the partition that it has been subscribed to one or more specified new partitions.
13 . The apparatus of claim 8 , wherein consumers of the topic share no state data.
14 . The apparatus of claim 8 , wherein:
said obtaining, said processing, and said processing are performed by a publisher within the publish-subscribe message system.
15 . A system, comprising:
one or more processors;
a repartition module comprising a non-transitory computer-readable medium storing instructions that, when executed, cause the system to:
during operation of a publish-subscribe message system that includes a message topic having N original partitions (N>1), reconfigure the topic to include multiple collections of partitions, wherein one of the multiple collections comprises the N original partitions;
a producer module comprising a non-transitory computer-readable medium storing instructions that, when executed, cause the system to:
obtain a first message for the topic, wherein the first message has an associated partition key;
process the partition key to identify a target collection of partitions of the topic; and
process the partition key to identify a destination partition in the target collection of partitions; and
a broker module comprising a non-transitory computer-readable medium storing instructions that, when executed, cause the system to:
store the first message in the destination partition of the target collection of partitions.
16 . The system of claim 15 , further comprising:
a consumer module comprising a non-transitory computer-readable medium storing instructions that, when executed, cause the system to:
after said configuring, but before said reconfiguring, subscribe to a first partition of the N original partitions; and
after said reconfiguring, subscribe to the first partition of each of the multiple collections of partitions.
17 . The system of claim 15 , wherein said repartitioning comprises:
creating one or more additional collections of N partitions; and
for each consumer subscribed to an i th partition in the original N partitions, subscribing the consumer to the i th partition in each of the one or more additional collections of N partitions.
18 . The system of claim 15 , wherein said repartitioning comprises:
adding, to each partition of the N original partitions, a message directing a consumer of the partition to subscribe to one or more specified new partitions.
19 . The system of claim 15 , wherein said repartitioning comprises:
adding, to each partition of the N original partitions, a message informing a consumer of the partition that it has been subscribed to one or more specified new partitions.
20 . The system of claim 15 , wherein consumers of the topic share no state data.