Systems and methods for storing message data
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, are described for storing message data in a PubSub system. In certain examples, messages are received from a plurality of publishers for a plurality of channels. The messages are stored in a writable portion of a respective buffer for the channel according to an order, wherein messages in the writable portion of the buffer are inaccessible to subscribers. The method may also include advancing a pointer demarcating a boundary between the writable portion and a readable portion of the buffer such that the message is in the readable portion after the pointer has advanced.
1. A method, comprising:
receiving messages on each of a plurality of channels;
storing, by one or more computer processors, each message of each of the plurality of channels in a writable portion of a respective buffer for the channel according to an order, wherein messages in the writable portion of the buffer are inaccessible to subscribers; and
advancing, by the one or more computer processors, a pointer demarcating a boundary between the writable portion and a readable portion of the buffer such that the message is in the readable portion after the pointer has advanced.
2. The method of claim 1 , further comprising:
allowing one or more subscribers to read from the readable portion of one or more of the buffers during the storing.
3. The method of claim 1 , wherein the pointer is advanced in an atomic operation.
4. The method of claim 3 , wherein the atomic operation cannot be interrupted by another process or thread of execution.
5. The method of claim 1 , wherein storing each message comprises:
storing a length of the message at a first location in the writable portion; and
storing the message in the writable portion following the first location.
6. The method of claim 1 , wherein advancing the pointer demarcating the boundary between the writable portion and the readable portion of the buffer comprises:
storing a sum of a length of the message and a current value of the pointer in the pointer.
7. The method of claim 1 , wherein each message is stored in the writable portion and moved to the readable portion before another message is stored in the writable portion.
8. The method of claim 1 , wherein each buffer for a particular channel expires at a different time based on a time-to-live for the buffer.
9. The method of claim 1 , wherein each buffer comprises a respective time-to-live upon expiration of which will cause the buffer to be inaccessible to publishers and subscribers.
10. The method of claim 1 , wherein the order comprises the order in which messages for the channel are received.
11. A system, comprising:
one or more computer processors programmed to perform operations to:
receive messages on each of a plurality of channels;
store each message of each of the plurality of channels in a writable portion of a respective buffer for the channel according to an order,
wherein messages in the writable portion of the buffer are inaccessible to subscribers; and
advance a pointer demarcating a boundary between the writable portion and a readable portion of the buffer such that the message is in the readable portion after the pointer has advanced.
12. The system of claim 11 , wherein the operations are further to:
allow one or more subscribers to read from the readable portion of one or more of the buffers during the store.
13. The system of claim 11 , wherein the pointer is advanced in an atomic operation.
14. The system of claim 11 , wherein to store each message the one or more computer processors are further to:
store a length of the message at a first location in the writable portion; and
store the message in the writable portion following the first location.
15. The system of claim 11 , wherein to advance the pointer demarcating the boundary between the writable portion and the readable portion of the buffer the one or more computer processors are further to:
store a sum of a length of the message and a current value of the pointer in the pointer.
16. The system of claim 11 , wherein each message is stored in the writable portion and moved to the readable portion before another message is stored in the writable portion.
17. The system of claim 11 , wherein each buffer for a particular channel expires at a different time based on a time-to-live for the buffer.
18. The system of claim 11 , wherein each buffer comprises a respective time-to-live upon expiration of which will cause the buffer to be inaccessible to publishers and subscribers.
19. The system of claim 11 , wherein the order comprises an order in which messages for the channel are received.
20. A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more computer processors, cause the one or more computer processors to:
receive messages on each of a plurality of channels;
store, by the one or more computer processors, each message of each of the plurality of channels in a writable portion of a respective buffer for the channel according to an order,
wherein messages in the writable portion of the buffer are inaccessible to subscribers; and
advance, by the one or more computer processors, a pointer demarcating a boundary between the writable portion and a readable portion of the buffer such that the message is in the readable portion after the pointer has advanced.