Systems and methods for storing message data
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, are described for storing message data in a PubSub system. In certain examples, the method includes storing messages of each of a plurality of channels in a writable portion of a respective buffer for the channel. The method may also include moving a pointer delineating a boundary between the writable portion and a readable portion of the buffer such that the messages are in the readable portion after the pointer has moved. The method may also include allowing one or more subscribers to read from the readable portion of one or more of the buffers during the storing.
1. A method, comprising:
storing messages of each of a plurality of channels in a writable portion of a respective buffer for the channel;
moving a pointer delineating a boundary between the writable portion and a readable portion of the buffer such that the messages are in the readable portion after the pointer has moved; and
allowing one or more subscribers to read from the readable portion of one or more of the buffers during the storing.
2. The method of claim 1 , comprising:
receiving the messages from a plurality of publishers on each of the plurality of channels.
3. The method of claim 1 , wherein messages in the writable portion of the buffer are inaccessible to subscribers.
4. The method of claim 1 , wherein the pointer is moved in an atomic operation.
5. The method of claim 4 , wherein the atomic operation cannot be interrupted by another process or thread of execution.
6. The method of claim 1 , wherein each message is stored in the writable portion and moved to the readable portion before another message is stored in the writable portion.
7. The method of claim 1 , wherein each buffer for a particular channel expires at a different time based on a time-to-live for the buffer.
8. The method of claim 1 , wherein each buffer comprises a respective time-to-live upon expiration of which will cause the buffer to be inaccessible to publishers and subscribers.
9. The method of claim 1 , wherein each message of each of the plurality of channels is stored in the respective buffer for the channel according to an order.
10. The method of claim 9 , wherein the order comprises the order in which messages for the channel are received.
11. A system, comprising:
one or more computer processors programmed to perform operations to:
store messages of each of a plurality of channels in a writable portion of a respective buffer for the channel;
move a pointer delineating a boundary between the writable portion and a readable portion of the buffer such that the messages are in the readable portion after the pointer has moved; and
allow one or more subscribers to read from the readable portion of one or more of the buffers during the storing.
12. The system of claim 11 , wherein the operations are further to:
receive the messages from a plurality of publishers on each of the plurality of channels.
13. The system of claim 11 , wherein messages in the writable portion of the buffer are inaccessible to subscribers.
14. The system of claim 11 , wherein the pointer is moved in an atomic operation.
15. The system of claim 11 , wherein each message is stored in the writable portion and moved to the readable portion before another message is stored in the writable portion.
16. The system of claim 11 , wherein each buffer for a particular channel expires at a different time based on a time-to-live for the buffer.
17. The system of claim 11 , wherein each buffer comprises a respective time-to-live upon expiration of which will cause the buffer to be inaccessible to publishers and subscribers.
18. The system of claim 11 , wherein each message of each of the plurality of channels is stored in the respective buffer for the channel according to an order.
19. The system of claim 18 , wherein the order comprises the order in which messages for the channel are received.
20. A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more computer processors, cause the one or more computer processors to:
store messages of each of a plurality of channels in a writable portion of a respective buffer for the channel;
move a pointer delineating a boundary between the writable portion and a readable portion of the buffer such that the messages are in the readable portion after the pointer has moved; and
allow one or more subscribers to read from the readable portion of one or more of the buffers during the storing.