Systems and methods for storing and transferring message data
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for storing and transferring messages. An example method includes storing a plurality of blocks in a queue, wherein each block includes one or more respective messages received from at least one publisher, wherein each block is associated with a time that the block was stored in the queue, and wherein storage times increase from a block designating a head of the queue to a block designating a tail of the queue. The example method also includes designating as inactive one or more blocks having associated storage times that are older than a first time, allowing messages to be read from inactive blocks until a second time that is older than the first time, and deleting one or more inactive blocks from the queue having associated storage times that are older than the second time.
1. A method, comprising:
performing by one or more computer processors:
storing a plurality of blocks in a queue,
wherein each block includes one or more respective messages received from at least one publisher,
wherein each block is associated with a time that the block was stored in the queue, and
wherein storage times increase from a block designating a head of the queue to a block designating a tail of the queue;
designating as inactive one or more blocks having associated storage times that are older than a first time;
allowing messages to be read from inactive blocks until a second time that is older than the first time; and
deleting one or more inactive blocks from the queue having associated storage times that are older than the second time.
2. The method of claim 1 , wherein the queue comprises a channel queue.
3. The method of claim 1 , wherein new blocks are stored at the tail of the queue.
4. The method of claim 1 , wherein messages from one or more blocks in the queue are read beginning at the head of the queue.
5. The method of claim 1 , wherein each block is assigned a time-to-reside (TTR), and wherein a block is designated as inactive when the TTR of the block expires.
6. The method of claim 5 , wherein allowing messages to be read from inactive blocks until the second time that is older than the first time comprises:
providing additional time for a subscriber to read messages from a block after the TTR for the block expires.
7. The method of claim 1 , wherein each block is assigned a time-to-live (TTL), and wherein a block is deleted when the TTL of the block expires.
8. The method of claim 1 , wherein the plurality of blocks are stored in the queue according to an order.
9. The method of claim 1 , wherein the messages are received from the at least one publisher for a single channel.
10. The method of claim 1 , comprising:
sending read messages to a respective subscriber.
11. A system, comprising:
one or more computer processors programmed to perform operations to:
store a plurality of blocks in a queue,
wherein each block includes one or more respective messages received from at least one publisher,
wherein each block is associated with a time that the block was stored in the queue, and
wherein storage times increase from a block designating a head of the queue to a block designating a tail of the queue;
designate as inactive one or more blocks having associated storage times that are older than a first time;
allow messages to be read from inactive blocks until a second time that is older than the first time; and
delete one or more inactive blocks from the queue having associated storage times that are older than the second time.
12. The system of claim 11 , wherein the queue comprises a channel queue.
13. The system of claim 11 , wherein new blocks are stored at the tail of the queue.
14. The system of claim 11 , wherein messages from one or more blocks in the queue are read beginning at the head of the queue.
15. The system of claim 11 , wherein each block is assigned a time-to-reside (TTR), and wherein a block is designated as inactive when the TTR of the block expires.
16. The system of claim 15 , wherein to allow messages to be read from inactive blocks until the second time the one or more computer processors are further to:
provide additional time for a subscriber to read messages from a block after the TTR for the block expires.
17. The system of claim 11 , wherein each block is assigned a time-to-live (TTL), and wherein a block is deleted when the TTL of the block expires.
18. The system of claim 11 , wherein the plurality of blocks are stored in the queue according to an order.
19. The system of claim 11 , wherein the messages are received from the at least one publisher for a single channel.
20. A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more computer processors, cause the one or more computer processors to:
store a plurality of blocks in a queue,
wherein each block includes one or more respective messages received from at least one publisher,
wherein each block is associated with a time that the block was stored in the queue, and
wherein storage times increase from a block designating a head of the queue to a block designating a tail of the queue;
designate as inactive one or more blocks having associated storage times that are older than a first time;
allow messages to be read from inactive blocks until a second time that is older than the first time; and
delete one or more inactive blocks from the queue having associated storage times that are older than the second time.