Large message support for a publish-subscribe messaging system

1. A method, comprising:

subscribing to a message stream brokered by a message brokering cluster that imposes a maximum message size, wherein messages in the message stream are identified by offsets assigned in accordance with an order of the messages;

responsive to receiving a given message of the message stream:

determining an offset to commit to the message brokering cluster, wherein the determined offset is based on whether one or more segments of one or more oversized messages are buffered; and

committing the determined offset to the message brokering cluster;

responsive to recovering from a crash, wherein the offset at which to resume the receipt of messages from the message stream is unknown, retrieving the determined offset from the message brokering cluster; and

resuming the receipt of messages from the message brokering cluster based on the determined offset;

wherein each of the one or more oversized messages has been divided into multiple segments and each of the multiple segments is sent as an individual message that complies with the maximum message size.

2. The method of claim 1 , wherein determining the offset to commit to the message brokering cluster comprises:

when the given message is a segment of a first oversized message:

when the given segment is the last segment of the first oversized message:

when at least one segment of another oversized message is buffered when the given message is received, selecting the offset of the oldest message that is buffered; and

when no segment of another oversized message is buffered when the given message is received, selecting the offset of a subsequent message; and

when the given segment is not the last segment of the first oversized message, selecting the offset of the oldest message that is buffered; and

when the given message is not a segment of an oversized message:

when at least one segment of an oversized message is buffered when the given message is received, selecting the offset of the oldest message that is buffered; and

when no segments of any oversized messages are buffered when the given message is received, selecting the offset of the subsequent message.

3. The method of claim 1 , further comprising, when the given message is a segment of a first oversized message and is not the last segment of the first oversized message:

buffering the given message; and

when the last segment of the first oversized message is received, rebuilding the first oversized message from the multiple segments and consuming the first oversized message.

4. The method of claim 3 , further comprising, when the given message is an initial segment of the first oversized message:

responsive to receiving a subsequent message, when a difference between the subsequent message's offset and the given message's offset exceeds a threshold value, deleting all buffered segments of the first oversized message, including the initial segment.

5. The method of claim 3 , wherein the first oversized message's offset is defined to be equal to the last segment's offset.

6. The method of claim 3 , wherein each of the segments of the first oversized message further comprises a segment identifier and a size of the first oversized message.

7. The method of claim 1 , further comprising:

recording a safe offset and a starting offset corresponding to the given message; and

responsive to receiving a request to seek to the given message's offset:

identifying a lower offset between the safe offset and the starting offset; and

seeking to a message corresponding to the lower offset.

8. The method of claim 1 , wherein:

the message brokering cluster corresponds to a Kafka cluster; and

the message stream corresponds to one of a Kafka topic and one or more partitions of a Kafka topic.

9. The method of claim 1 , wherein each message comprises data formatted in one of:

JavaScript Object Notation; and

Avro.

10. An apparatus, comprising:

one or more processors; and

memory storing instructions that, when executed by the one or more processors, cause the apparatus to:

subscribe to a message stream brokered by a message brokering cluster that imposes a maximum message size, wherein messages in the message stream are identified by offsets assigned in accordance with an order of the messages;

responsive to receiving a given message of the message stream:

determine an offset to commit to the message brokering cluster, wherein the determined offset is based on whether one or more segments of one or more oversized messages are buffered; and

commit the determined offset to the message brokering cluster;

responsive to recovering from a crash, wherein the offset at which to resume the receipt of messages from the message stream is unknown, retrieve the determined offset from the message brokering cluster; and

resume the receipt of messages from the message brokering cluster based on the determined offset;

wherein each of the one or more oversized messages has been divided into multiple segments and each of the multiple segments is sent as an individual message that complies with the maximum message size.

11. The apparatus of claim 10 , wherein determining the offset to commit to the message brokering cluster comprises:

when the given message is a segment of a first oversized message:

when the given segment is the last segment of the first oversized message:

when at least one segment of another oversized message is buffered when the given message is received, selecting the offset of the oldest message that is buffered; and

when no segment of another oversized message is buffered when the given message is received, selecting the offset of a subsequent message; and

when the given segment is not the last segment of the first oversized message, selecting the offset of the oldest message that is buffered; and

when the given message is not a segment of an oversized message:

when at least one segment of an oversized message is buffered when the given message is received, selecting the offset of the oldest message that is buffered; and

when no segments of any oversized messages are buffered when the given message is received, selecting the offset of the subsequent message.

12. The apparatus of claim 10 , wherein the apparatus is further caused to, when the given message is a segment of a first oversized message and is not the last segment of the first oversized message:

buffer the given message; and

when the last segment of the first oversized message is received, rebuild the first oversized message from the multiple segments and consume the first oversized message.

13. The apparatus of claim 12 , wherein the apparatus is further caused to, when the given message is an initial segment of the first oversized message:

responsive to receiving a subsequent message, when a difference between the subsequent message's offset and the given message's offset exceeds a threshold value, delete all buffered segments of the first oversized message, including the initial segment.

14. The apparatus of claim 12 , wherein the first oversized message's offset is defined to be equal to the last segment's offset.

15. The apparatus of claim 12 , wherein each of the segments of the first oversized message further comprises a segment identifier and a size of the first oversized message.

16. The apparatus of claim 10 , wherein the apparatus is further caused to:

record a safe offset and a starting offset corresponding to the given message; and

responsive to receiving a request to seek to the given message's offset:

identify a lower offset between the safe offset and the starting offset; and

seek to a message corresponding to the lower offset.

17. The apparatus of claim 10 , wherein:

the message brokering cluster corresponds to a Kafka cluster; and

the message stream corresponds to one of a Kafka topic and one or more partitions of a Kafka topic.

18. The apparatus of claim 10 , wherein each message comprises data formatted in one of:

JavaScript Object Notation; and

Avro.

19. A system, comprising:

one or more processors;

a message brokering module comprising a non-transitory computer-readable medium storing instructions that, when executed, cause the system to broker a message stream and impose a maximum message size on the message stream, wherein messages in the message stream are identified by offsets assigned in accordance with an order of the messages; and

a message consuming module comprising a non-transitory computer-readable medium storing instructions that, when executed, cause the system to:

responsive to receiving a given message of the message stream:

determine an offset to commit to a message brokering cluster, wherein the determined offset is based on whether one or more segments of one or more oversized messages are buffered; and

commit the determined offset to the message brokering cluster;

responsive to recovering from a crash, wherein the offset at which to resume the receipt of messages from the message stream is unknown, retrieve the determined offset from the message brokering cluster; and

resume the receipt of messages from the message brokering cluster based on the determined offset;

wherein each of the one or more oversized messages has been divided into multiple segments and each of the multiple segments is sent as an individual message that complies with the maximum message size.

20. The system of claim 19 , wherein determining the offset to commit to the message brokering cluster comprises:

when the given message is a segment of a first oversized message:

when the given segment is the last segment of the first oversized message:

when at least one segment of another oversized message is buffered when the given message is received, selecting the offset of the oldest message that is buffered; and

when no segment of another oversized message is buffered when the given message is received, selecting the offset of a subsequent message; and

when the given segment is not the last segment of the first oversized message, selecting the offset of the oldest message that is buffered; and

when the given message is not a segment of an oversized message:

when at least one segment of an oversized message is buffered when the given message is received, selecting the offset of the oldest message that is buffered; and

when no segments of any oversized messages are buffered when the given message is received, selecting the offset of the subsequent message.