Message compression in scalable messaging system

1. A computer-implemented method comprising:

receiving from a plurality of publisher clients a plurality of messages, each message being for a channel of a plurality of channels wherein each channel comprises an ordered plurality of messages;

encoding, by one or more computer processors, each message for a first channel from the plurality of channels based on a dictionary for the first channel, the dictionary defining a pattern associated with each message for the first channel, wherein encoding each message for the first channel comprises compressing the message for the first channel according to the pattern;

storing encoded messages for the first channel in one or more respective buffers according to the order, each buffer having a respective time-to-live and residing on a respective node;

retrieving encoded messages for the first channel from respective buffers having time-to-lives that have not expired and according to the order;

inspecting content of at least one retrieved encoded message;

determining, by the one or more computer processors, from the content at least one pattern used to encode each retrieved encoded message;

decoding each retrieved encoded message based on the dictionary and the determined pattern, wherein decoding comprises decompressing each message in the first channel according to the determined pattern; and

sending the decoded messages to a plurality of subscriber clients.

2. The method of claim 1 wherein the pattern is shared by at least some of the plurality of messages for the first channel.

3. The method of claim 1 wherein the pattern comprises a text string.

4. The method of claim 1 wherein the pattern corresponds to a common data field shared by at least some of the plurality of messages for the first channel.

5. The method of claim 1 wherein the pattern comprises a data type.

6. The method of claim 1 further comprising: adding the determined pattern to the dictionary for the first channel.

7. The method of claim 1 wherein storing encoded messages for the first channel in one or more respective buffers comprises: sending a plurality of encoded messages to a first buffer on a first node, wherein the first node stores the plurality of encoded messages in a first block of one or more blocks within the first buffer, wherein each block comprises a respective time-to-live.

8. The method of claim 7 wherein retrieving encoded messages for the first channel comprises retrieving encoded messages from one or more of the one or more blocks within the first buffer having respective time-to-lives that have not expired.

9. A system comprising:

a memory; and

one or more computer processors, operatively coupled with the memory, programmed to perform operations to:

receive from a plurality of publisher clients a plurality of messages, each message being for a channel of a plurality of channels wherein each channel comprises an ordered plurality of messages;

encode each message for a first channel from the plurality of channels based on a dictionary for the first channel, the dictionary defining a pattern associated with each message for the first channel, wherein encoding each message for the first channel comprises compressing the message for the first channel according to the pattern;

store encoded messages for the first channel in one or more respective buffers according to the order, each buffer having a respective time-to-live and residing on a respective node;

retrieve encoded messages for the first channel from respective buffers having time-to-lives that have not expired and according to the order;

inspect content of at least one retrieved encoded message;

determine from the content at least one pattern used to encode each retrieved encoded message;

decode each retrieved encoded message based on the dictionary and the determined pattern, wherein decoding comprises decompressing each message in the first channel according to the determined pattern; and

send the decoded messages to a plurality of subscriber clients.

10. The system of claim 9 wherein the pattern is shared by at least some of the plurality of messages for the first channel.

11. The system of claim 9 wherein the pattern comprises a text string.

12. The system of claim 9 wherein the pattern corresponds to a common data field shared by at least some of the plurality of messages for the first channel.

13. The system of claim 9 wherein the pattern comprises a data type.

14. The system of claim 9 , the operations further to: add the determined pattern to the dictionary for the first channel.

15. The system of claim 9 wherein to store the encoded messages for the first channel in one or more respective buffers the operations are to: send a plurality of encoded messages to a first buffer on a first node, wherein the first node stores the plurality of encoded messages in a first block of one or more blocks within the first buffer wherein each block comprises a respective time-to-live.

16. The system of claim 15 wherein to retrieve the encoded messages for the first channel, the operations are to: retrieve the encoded messages from one or more of the one or more blocks within the first buffer having respective time-to-lives that have not expired.

17. A non-transitory machine-readable medium having instructions stored thereon that, when executed by one or more computer processors, cause the one or more computer processors:

receive from a plurality of publisher clients a plurality of messages, each message being for a channel of a plurality of channels wherein each channel comprises an ordered plurality of messages;

encode each message for a first channel from the plurality of channels based on a dictionary for the first channel, the dictionary defining a pattern associated with each message for the first channel, wherein encoding each message for the first channel comprises compressing the message for the first channel according to the pattern;

store encoded messages for the first channel in one or more respective buffers according to the order, each buffer having a respective time-to-live and residing on a respective node;

retrieve encoded messages for the first channel from respective buffers having time-to-lives that have not expired and according to the order;

inspect content of at least one retrieved encoded message;

determine from the content at least one pattern used to encode each retrieved encoded message;

decode each retrieved encoded message based on the dictionary and the determined pattern, wherein decoding comprises decompressing each message in the first channel according to the determined pattern; and

send the decoded messages to a plurality of subscriber clients.

18. The non-transitory machine-readable medium of claim 17 wherein the pattern is shared at least some of the plurality of messages for the first channel.

19. The non-transitory machine-readable medium of claim 17 wherein the pattern comprises a text string.

20. The non-transitory machine-readable medium of claim 17 wherein the pattern corresponds to a common data field shared by at least some of the plurality of messages for the first channel.

21. The non-transitory machine-readable medium of claim 17 wherein the pattern comprises a data type.

22. The non-transitory machine-readable medium of claim 17 wherein the one or more computer processors are further to: add the determined pattern to the dictionary for the first channel.

23. The non-transitory machine-readable medium of claim 17 wherein to store the encoded messages for the first channel in one or more respective buffers the one or more computer processors are to:

send a plurality of encoded messages to a first buffer on a first node, wherein the first node stores the plurality of encoded messages in a first block of one or more blocks within the first buffer, wherein each block comprises a respective time-to-live.

24. The non-transitory machine-readable medium of claim 23 wherein to retrieve the encoded messages for the first channel, the one or more computer processors are to: retrieve the encoded messages from one or more of the one or more blocks within the first buffer having respective time-to-lives that have not expired.