Message broker system with parallel persistence

1. A method for a message broker computer system to process a message, the method comprising:

a master broker computer system receiving a message from a client producer;

the master broker computer system, responsive to receiving the message from the client producer, generating a message event, the message event uniquely identifying the message with an epoch value associated with the master broker computer system and a sequence number associated with the message;

the master broker computer system distributing the message event to a plurality of event stores and a plurality of slave broker computer systems, each event store comprising an event store associated persistent storage system and each slave broker computer system comprising both a slave broker associated persistent storage system and an associated memory, wherein each event store stores the message event in the event store associated persistent storage system, and wherein each slave broker computer system holds the message event in the associated memory;

the master broker computer system, responsive to receiving persistence notifications from a quorum of event stores in the plurality of event stores, determining that the message event is stabilized, wherein each persistence notification indicates that the message event has been stored in the associated persistent storage system of the respective event store and wherein the quorum comprises at least two event stores;

the master broker computer system, responsive to determining that the message event is stabilized, storing the message event in an associated persistent storage and sending a notification to the client producer;

the master broker computer system generating a second message event including the epoch value and the sequence number to indicate that the message event has been stabilized; and

the master broker computer system distributing the second message event to the plurality of event stores and the plurality of slave broker computer systems, wherein the second message event causes each slave broker computer system to transfer the message event from its respective associated memory to its respective persistent storage system.

2. The method of claim 1 , further comprising:

storing a broker computer system state for each message broker computer system, wherein a broker computer system state is a representation of the message event stream that has been persisted at the broker computer system.

3. The method of claim 2 , further comprising updating the broker computer system state of each message broker computer system on receiving a message event from the master broker computer system, wherein updating the broker computer system state includes performing at least one of removal of a message event from the persistent storage or overwriting the message event with a received message event from the master broker computer system.

4. The method of claim 1 , further comprising the master broker computer system removing the message event from an internal memory of the master broker computer system once the message event is stabilized.

5. The method of claim 1 , wherein each slave broker system is configured to remove the message event from its respective persistent storage system system on receiving at least one of a second message event from the master broker computer system or a stability message event from the master broker computer system, wherein a stability message event includes information associated with a last known stable message event at the master broker computer system.

6. The method of claim 1 , wherein a second message event includes a message event, a sequence number associated with the last stable message event, an epoch value associated with the master broker computer system that sent the last stable message event, a sequence number associated with the message event and an epoch value associated with the master broker computer system.

7. The method of claim 1 , wherein the master broker computer system distributes the message event to the plurality of slave broker computer systems and the plurality of event stores in parallel.

8. A master broker computer system comprising:

one or more processors; and

one or more memories operatively coupled to at least one of the one or more processors and having instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to:

receive a message from a client producer;

responsive to receiving the message from the client producer, generate a message event, the message event uniquely identifying the message with an epoch value associated with the master broker computer system and a sequence number associated with the message;

distribute the message event to a plurality of event stores and a plurality of slave broker computer systems, each event store comprising an event store associated persistent storage system and each slave broker computer system comprising both a slave broker associated persistent storage system and an associated memory, wherein each event store stores the message event in the event store associated persistent storage system, and wherein each slave broker computer system holds the message event in the associated memory;

responsive to receiving persistence notifications from a quorum of event stores in the plurality of event stores, determine that the message event is stabilized, wherein each persistence notification indicates that the message event has been stored in the associated persistent storage system of the respective event store and wherein the quorum comprises at least two event stores;

responsive to determining that the message event is stabilized, store the message event in an associated persistent storage and send a notification to the client producer;

generate a second message event including the epoch value and the sequence number to indicate that the message event has been stabilized; and

distribute the second message event to the plurality of event stores and the plurality of slave broker computer systems, wherein the second message event causes each slave broker computer system to transfer the message event from its respective associated memory to its respective persistent storage system.

9. The apparatus of claim 8 , wherein at least one of the one or more memories has further instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to:

store a broker computer system state for each message broker computer system, wherein a broker computer system state is a representation of the message event stream that has been persisted at the broker computer system.

10. The apparatus of claim 9 , wherein at least one of the one or more memories has further instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to:

update the broker computer system state of each message broker computer system on receiving a message event from the master broker computer system, wherein updating the broker computer system state includes performing at least one of removal of a message event from the persistent storage or overwriting the message event with a received message event from the master broker computer system.

11. The apparatus of claim 8 , wherein at least one of the one or more memories has further instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to:

remove the message event from an internal memory of the master broker computer system once the message event is stabilized.

12. The apparatus of claim 8 , wherein each slave broker system is configured to remove the message event from its respective persistent storage system on receiving at least one of a second message event from the master broker computer system or a stability message event from the master broker computer system, wherein a stability message event includes information associated with a last known stable message event at the master broker computer system.

13. The apparatus of claim 8 , wherein a second message event includes a message event, a sequence number associated with the last stable message event, an epoch value associated with the master broker computer system that sent the last stable message event, a sequence number associated with the message event and an epoch value associated with the master broker computer system.

14. The apparatus of claim 8 , wherein the master broker computer system distributes the message event to the plurality of slave broker computer systems and the plurality of event stores in parallel.

15. At least one non-transitory computer-readable medium storing computer-readable instructions that, when executed by a master broker computer system, causes the master broker computer system to:

receive a message from a client producer;

responsive to receiving the message from the client producer, generate a message event, the message event uniquely identifying the message with an epoch value associated with the master broker computer system and a sequence number associated with the message;

distribute the message event to a plurality of event stores and a plurality of slave broker computer systems, each event store comprising an event store associated persistent storage system and each slave broker computer system comprising both a slave broker associated persistent storage system and an associated memory, wherein each event store stores the message event in the event store associated persistent storage system, and wherein each slave broker computer system holds the message event in the associated memory;

responsive to receiving persistence notifications from a quorum of event stores in the plurality of event stores, determine that the message event is stabilized, wherein each persistence notification indicates that the message event has been stored in the associated persistent storage system of the respective event store and wherein the quorum comprises at least two event stores;

responsive to determining that the message event is stabilized, store the message event in an associated persistent storage and send a notification to the client producer;

generate a second message event including the epoch value and the sequence number to indicate that the message event has been stabilized; and

distribute the second message event to the plurality of event stores and the plurality of slave broker computer systems, wherein the second message event causes each slave broker computer system to transfer the message event from its respective i4s associated memory to its respective persistent storage system.

16. The at least one non-transitory computer-readable medium of claim 15 , further storing computer-readable instructions that, when executed by at least one of the one or more computing devices, cause at least one of the one or more computing devices to:

store a broker computer system state for each message computer system, wherein a broker computer system state is a representation of the message event stream that has been persisted at the broker computer system.

17. The at least one non-transitory computer-readable medium of claim 16 , further storing computer-readable instructions that, when executed by at least one of the one or more computing devices, cause at least one of the one or more computing devices to:

update the broker computer system state of each message broker computer system on receiving a message event from the master broker computer system, wherein updating the broker computer system state includes performing at least one of removal of a message event from the persistent storage or overwriting the message event with a received message event from the master broker computer system.

18. The at least one non-transitory computer-readable medium of claim 15 , further storing computer-readable instructions that, when executed by at least one of the one or more computing devices, cause at least one of the one or more computing devices to:

remove the message event from an internal memory of the master broker computer system once the message event is stabilized.

19. The at least one non-transitory computer-readable medium of claim 15 , wherein each slave broker system is configured to remove the message event from its respective persistent storage system system on receiving at least one of a second message event from the master broker computer system or a stability message event from the master broker computer system, wherein a stability message event includes information associated with a last known stable message event at the master broker computer system.

20. The at least one non-transitory computer-readable medium of claim 15 , wherein a second message event includes a message event, a sequence number associated with the last stable message event, an epoch value associated with the master broker computer system that sent the last stable message event, a sequence number associated with the message event and an epoch value associated with the master broker computer system.

21. The at least one non-transitory computer-readable medium of claim 15 , wherein the master broker computer system distributes the message event to the plurality of slave broker computer systems and the plurality of event stores in parallel.