De-multiplexing streaming data from a fast write ahead durable log into segments
Implementations are provided herein for managing streaming data that appended and multiplexed into a durable write ahead replicated log. By writing data in the durable log, large amounts of small writes can be processed quickly. Data in the durable log can be de-multiplexed and packaged into segment containers. Segment containers can serialize stream segment specific data and can be stored in long term storage. Data that has been stored in long term storage can be truncated from the durable write ahead log making room for new data.
1. A method comprising:
maintaining a set of segment containers, wherein each segment container in the set of segment containers is associated with at least one segment, and each segment is associated with one segment container;
assigning operations received from a client application to segment containers within the set of segment containers using a predefined mapping function;
reading a set of operations from a durable log in a segment container, wherein the durable log contains multiplexed data associated with a set of stream segments that are mapped to the segment container;
de-multiplexing the set of operations into a set of stream segment buffers based on a stream segment associated with each operation in the set of operations in the durable log;
packaging data within a segment among the set of segments, within a segment container among the set of segment containers, based on the set of stream segment buffers, in response to a policy trigger; and
storing the packaged data of the segment in a long-term storage location.
2. The method of claim 1 , wherein the durable log is replicated and stored in non-volatile storage.
3. The method of claim 2 , wherein the long-term storage location has a higher latency, a higher throughput, and a lower cost per gigabyte than the durable log.
4. The method of claim 1 , wherein the policy trigger is based on at least one of a size of outstanding accumulated date for the segment in the segment's buffer or a last received operation time associated with the segment in the segment's buffer.
5. The method of claim 1 , further comprising:
truncating the durable log based on the packaged data.
6. A system comprising at least one storage device and at least one hardware processor configured to:
maintain a set of segment containers, wherein each segment container in the set of segment containers is associated with at least one segment, and each segment is associated with one segment container;
assign operations received from a client application to segment containers within the set of segment containers using a predefined mapping function;
read a set of operations from a durable log in a segment container, wherein the durable log contains multiplexed data associated with a set of stream segments that are mapped to the segment container;
de-multiplex the set of operations into a set of stream segment buffers based on a stream segment associated with each operation in the set of operations in the durable log;
package data within a segment among the set of segments, within a segment container among the set of segment containers, based on the set of stream segment buffers, in response to a policy trigger; and
store the packaged data of the segment in a long-term storage location.
7. The system of claim 6 , wherein the durable log is replicated and stored in non-volatile storage.
8. The system of claim 7 , wherein the long-term storage location has a higher latency, a higher throughput, and a lower cost per gigabyte than the durable log.
9. The system of claim 6 , wherein the policy trigger is based on at least one of a size of outstanding accumulated date for the segment in the segment's buffer or a last received operation time associated with the segment in the segment's buffer.
10. The system of claim 6 , further configured to:
truncate the durable log based on the packaged data.
11. A non-transitory computer readable medium with program instructions stored thereon, when executed by a processor, cause the processor to perform the following acts:
maintaining a set of segment containers, wherein each segment container in the set of segment containers is associated with at least one segment, and each segment is associated with one segment container;
assigning operations received from a client application to segment containers within the set of segment containers using a predefined mapping function;
reading a set of operations from a durable log in a segment container, wherein the durable log contains multiplexed data associated with a set of stream segments that are mapped to the segment container;
de-multiplexing the set of operations into a set of stream segment buffers based on a stream segment associated with each operation in the set of operations in the durable log;
packaging data within a segment among the set of segments, within a segment container among the set of segment containers, based on the set of stream segment buffers, in response to a policy trigger; and
storing the packaged data of the segment in a long-term storage location.
12. The non-transitory computer readable medium of claim 11 , wherein the durable log is replicated and stored in non-volatile storage.
13. The non-transitory computer readable medium of claim 12 , wherein the long-term storage location has a higher latency, a higher throughput, and a lower cost per gigabyte than the durable log.
14. The non-transitory computer readable medium of claim 11 , wherein the policy trigger is based on at least one of a size of outstanding accumulated date for the segment in the segment's buffer or a last received operation time associated with the segment in the segment's buffer.
15. The non-transitory computer readable medium of claim 11 , with program instructions stored thereon to further perform the following acts:
truncating the durable log based on the packaged data.