Selective distribution of messages in a scalable, real-time messaging system
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for selective distribution of messages from a scalable, real-time messaging system. An example method includes: receiving query instructions from a subscriber client, wherein the subscriber client is subscribed to a channel of a plurality of channels; receiving a plurality of messages from a plurality of publisher clients, wherein each message is associated with a particular one of the plurality of channels; applying the query instructions to the plurality of messages for the channel as the messages are received; and sending the messages resulting from the application of the query instructions to the subscriber client, wherein the messages received by the subscriber client from the channel are limited to those that satisfy the query instructions.
1. A computer-implemented method, comprising:
receiving query instructions from a subscriber client, wherein the subscriber client is subscribed to a channel of a plurality of channels;
receiving a plurality of messages from a plurality of publisher clients, wherein each message is associated with a particular one of the plurality of channels;
applying, by one or more computer processors, the query instructions to the plurality of messages for the channel as the messages are received; and
sending the messages resulting from the application of the query instructions to the subscriber client, wherein the messages received by the subscriber client from the channel are limited to those that satisfy the query instructions.
2. The method of claim 1 , wherein applying the query instructions to the plurality of messages for the channel as the messages are received comprises: filtering the messages based on one or more publisher clients.
3. The method of claim 1 , wherein applying the query instructions to the plurality of messages for the channel as the messages are received comprises: applying one or more of a filter, a function, or an aggregation to the plurality of messages for the channel as the messages are received.
4. The method of claim 1 , comprising:
building an index based on the query instructions; and
using the index when applying the query instructions to the plurality of messages for the channel as the messages are received.
5. The method of claim 1 , comprising: maintaining a query frequency, wherein the query frequency describes a number of times the query instructions refer to a particular data element within the messages.
6. The method of claim 5 , comprising: storing an index if the query frequency exceeds a threshold, otherwise discarding the index after the index is applied for a defined period.
7. The method of claim 4 , comprising: applying the query instructions to non-indexed messages received prior to receipt of the query instructions.
8. The method of claim 7 , comprising: combining results of applying the query instructions to the plurality of messages as the messages are received with results of applying the query instructions to non-indexed messages received prior to receipt of the query instructions.
9. The method of claim 1 , wherein the query instructions from the subscriber client include a request for a history of messages for the channel.
10. The method of claim 9 , wherein applying the query instructions to the plurality of messages for the channel as the messages are received comprises: applying one or more of a filter, a function, and an aggregation to the history of messages for the channel.
11. A system, comprising:
a memory; and
one or more computer processors, operatively coupled with the memory, programmed to perform operations to:
receive query instructions from a subscriber client, wherein the subscriber client is subscribed to a channel of a plurality of channels;
receive a plurality of messages from a plurality of publisher clients, wherein each message is associated with a particular one of the plurality of channels;
apply the query instructions to the plurality of messages for the channel as the messages are received; and
send the messages resulting from the application of the query instructions to the subscriber client, wherein the messages received by the subscriber client from the channel are limited to those that satisfy the query instructions.
12. The system of claim 11 , wherein to apply the query instructions to the plurality of messages for the channel as the messages are received the one or more computer processors are to:
apply one or more of a filter, a function, or an aggregation to the plurality of messages for the channel as the messages are received.
13. The system of claim 11 , the operations further to:
build an index based on the query instructions; and
utilize the index to apply the query instructions to the plurality of messages for the channel as the messages are received.
14. The system of claim 11 , the operations further to:
maintain a query frequency, wherein the query frequency describes a number of times the query instructions refer to a particular data element within the messages.
15. The system of claim 14 , wherein the operations further to:
store an index if the query frequency exceeds a threshold, otherwise discarding the index after the index is applied for a defined period.
16. The system of claim 13 , the operations further to:
apply the query instructions to non-indexed messages received prior to receipt of the query instructions.
17. The system of claim 16 , the operations further to:
combine results of applying the query instructions to the plurality of messages as the messages are received with results of applying the query instructions to non-indexed messages received prior to receipt of the query instructions.
18. The system of claim 11 , wherein the query instructions from the subscriber client include a request for a history of messages for the channel.
19. The system of claim 18 , wherein to apply the query instructions to the plurality of messages for the channel as the messages are received the one or more computer processors are to:
apply one or more of a filter, a function, or an aggregation to the history of messages for the channel.
20. A non-transitory machine-readable medium having instructions stored thereon that, when executed by one or more computer processors, cause the one or more computer processors to:
receive query instructions from a subscriber client, wherein the subscriber client is subscribed to a channel of a plurality of channels;
receive a plurality of messages from a plurality of publisher clients, wherein each message is associated with a particular one of the plurality of channels;
apply, by the one or more computer processors, the query instructions to the plurality of messages for the channel as the messages are received; and
send the messages resulting from the application of the query instructions to the subscriber client, wherein the messages received by the subscriber client from the channel are limited to those that satisfy the query instructions.