Message broker consumer group versioning
In an example embodiment a mechanism for consumer group versioning is introduced. Here, each application runtime provides a version for any consumer group during its deployment and keeps increasing the version whenever there is an enhancement or bug fix. Thus, both the application and the consumer group will have a version. Once it is recognized that a consumer group assigned to partitions in a topic has an outdated consumer group version number (i.e., a consumer group with the same name/application but a later consumer group version number has been registered with the message broker), the old consumer group is disconnected immediately. This allows the message broker to immediately assign partitions to the consumers in the newer consumer group, thus avoiding the aforementioned delays and associated technical problems.
1. A system comprising:
at least one hardware processor; and
a computer-readable medium storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising:
receiving a connection request sent from a first application instance of an application to a message broker, the connection request including, for a topic of the message broker, a consumer group identification and a consumer group version number, the consumer group identification identifying a group of message consumers and the consumer group version number indicating a version of the group of message consumers, the consumer group version number being different than a version number assigned to an application to which the group of message consumers is associated, each message consumer of the group of message consumers being microservices;
determining whether the consumer group version number is greater than a last known consumer group version number for the consumer group;
in response to a determination that the consumer group version number is greater than the last known consumer group version number for the consumer group, causing the message broker to disconnect one or more message consumers, including a second application instance of the application, having an older consumer group version number than the consumer group version number of the first application instance; and
causing assignment of one or more message consumers in a consumer group associated with the consumer group identification to one or more partitions of the topic, each partition responsible for serving a different subset of messages posted to the topic, each partition only assigned to a single message consumer at a time, such that messages posted to the topic are served by the message broker to one or more of the one or more message consumers associated with the consumer group identification.
2. The system of claim 1 , wherein the receiving, determining, causing the message broker to disconnect, and the causing assignment are performed by the message broker.
3. The system of claim 1 , wherein the receiving, determining, causing the message broker to disconnect, and the causing assignment are performed by an extension of the message broker.
4. The system of claim 1 , wherein the receiving, determining, causing the message broker to disconnect, and the causing assignment are performed by a component separate and distinct from the message broker.
5. The system of claim 4 , wherein the causing the message broker to disconnect includes sending a command to the message broker to perform the disconnecting.
6. The system of claim 1 , wherein the first application instance is a microservice.
7. The system of claim 1 , wherein the causing assignment includes using a partitioning scheme.
8. A method comprising:
receiving a connection request sent from a first application instance of an application to a message broker, the connection request including, for a topic of the message broker, a consumer group identification and a consumer group version number, the consumer group identification identifying a group of message consumers and the consumer group version number indicating a version of the group of message consumers, the consumer group version number being different than a version number assigned to an application to which the group of message consumers is associated, each message consumer of the group of message consumers being microservices;
determining whether the consumer group version number is greater than a last known consumer group version number for the consumer group;
in response to a determination that the consumer group version number is greater than the last known consumer group version number for the consumer group, causing the message broker to disconnect one or more message consumers, including a second application instance of the application, having an older consumer group version number than the consumer group version number of the first application instance; and
causing assignment of one or more message consumers in a consumer group associated with the consumer group identification to one or more partitions of the topic, each partition responsible for serving a different subset of messages posted to the topic, each partition only assigned to a single message consumer at a time, such that messages posted to the topic are served by the message broker to one or more of the one or more message consumers associated with the consumer group identification.
9. The method of claim 8 , wherein the receiving, determining, causing the message broker to disconnect, and the causing assignment are performed by the message broker.
10. The method of claim 8 , wherein the receiving, determining, causing the message broker to disconnect, and the causing assignment are performed by an extension of the message broker.
11. The method of claim 8 , wherein the receiving, determining, causing the message broker to disconnect, and the causing assignment are performed by a component separate and distinct from the message broker.
12. The method of claim 11 , wherein the causing the message broker to disconnect includes sending a command to the message broker to perform the disconnecting.
13. The method of claim 8 , wherein the first application instance is a microservice.
14. The method of claim 8 , wherein the causing assignment includes using a partitioning scheme.
15. A non-transitory machine-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving a connection request sent from a first application instance of an application to a message broker, the connection request including, for a topic of the message broker, a consumer group identification and a consumer group version number, the consumer group identification identifying a group of message consumers and the consumer group version number indicating a version of the group of message consumers, the consumer group version number being different than a version number assigned to an application to which the group of message consumers is associated, each message consumer of the group of message consumers being microservices;
determining whether the consumer group version number is greater than a last known consumer group version number for the consumer group;
in response to a determination that the consumer group version number is greater than the last known consumer group version number for the consumer group, causing the message broker to disconnect one or more message consumers, including a second application instance of the application, having an older consumer group version number than the consumer group version number of the first application instance; and
causing assignment of one or more message consumers in a consumer group associated with the consumer group identification to one or more partitions of the topic, each partition responsible for serving a different subset of messages posted to the topic, each partition only assigned to a single message consumer at a time, such that messages posted to the topic are served by the message broker to one or more of the one or more message consumers associated with the consumer group identification.
16. The non-transitory machine-readable medium of claim 15 , wherein the receiving, determining, causing the message broker to disconnect, and the causing assignment are performed by the message broker.
17. The non-transitory machine-readable medium of claim 15 , wherein the receiving, determining, causing the message broker to disconnect, and the causing assignment are performed by an extension of the message broker.
18. The non-transitory machine-readable medium of claim 15 , wherein the receiving, determining, causing the message broker to disconnect, and the causing assignment are performed by a component separate and distinct from the message broker.
19. The non-transitory machine-readable medium of claim 18 , wherein the causing the message broker to disconnect includes sending a command to the message broker to perform the disconnecting.
20. The non-transitory machine-readable medium of claim 15 , wherein the first application instance is a microservice.