Forking data streams in a streaming data storage platform
The technology describes forking (dividing) a data stream into multiple data streams comprising appending a first new data stream and a second new data stream to the original data stream. After forking, the two new data streams have different sets of stream fragments, with one or more of the older stream fragments common to both sets. Data events before the forking operation are read from the original data stream, while data events appended after the forking operation are appended to and read from only one of the data streams, based on which stream each writer/reader is registered to. Also described is truncating the original data stream based on one stream identifier, while leaving the original data intact for another stream identifier. Only when no stream needs a portion of the original data (resulting from separate truncation operations) is the data portion deleted.
1. A system, comprising:
a processor; and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, the operations comprising:
maintaining first stream fragment metadata and first stream head metadata for a first data stream;
creating a second data stream based on the first data stream, comprising setting second stream fragment metadata equal to the first stream fragment metadata, and setting second stream head metadata equal to the first stream head metadata;
creating a new first stream fragment at a tail of the first data stream; and
creating a new second stream fragment at the tail of the first data stream.
2. The system of claim 1 , wherein the operations further comprise closing an epoch prior to creating the new first stream fragment.
3. The system of claim 1 , wherein the operations further comprise closing an epoch after creating the new first stream fragment.
4. The system of claim 1 , wherein the first stream head metadata comprises a first stream identifier and a pointer to a location in the first data stream.
5. The system of claim 4 , wherein the operations further comprise modifying the first stream head metadata of the first data stream in response to a truncation request, the modifying the first stream head metadata comprising moving the pointer to a more forward location in the first data stream.
6. The system of claim 5 , wherein the operations further comprise determining that no data stream created from the first data stream contains data prior to the more forward location, and in response to the determining, deleting the data of the first data stream that is prior to the more forward position.
7. The system of claim 1 , wherein the operations further comprise registering a program to read from the second data stream.
8. The system of claim 1 , wherein the operations further comprise,
appending events to the second stream fragment resulting in a new tail of the second data stream,
creating a third data stream based on the second data stream, comprising setting third stream fragment metadata to equal the second stream fragment metadata, and setting third stream head metadata to equal the second stream head metadata,
creating a new second stream fragment at the new tail of the second data stream, and
creating a new third stream fragment at the new tail of the second data stream.
9. The system of claim 1 , wherein the first stream fragment metadata comprises a series of stream fragments.
10. The system of claim 1 , wherein the first stream fragment data to which a stream fragment corresponds comprises a sequence of epochs.
11. A method, comprising:
dividing, by a processor of a streaming data storage system, an existing data stream into first data stream and a second data stream, the dividing comprising:
for the first data stream, inheriting stream fragment metadata of the existing data stream as first stream fragment metadata, inheriting stream head metadata of the existing data stream as first stream head metadata, creating a new first stream fragment at a tail of the existing data stream, and updating the first stream head metadata based on the new first stream fragment; and
for the second data stream, inheriting stream fragment metadata of the existing data stream as second stream fragment metadata, inheriting stream head metadata of the existing data stream as second stream head metadata, creating a new second stream fragment at a tail of the existing data stream, and updating the second stream head metadata based on the new second stream fragment.
12. The method of claim 11 , further comprising inheriting, by the first data stream, an identifier of the existing data stream, and assigning, to the second data stream, a new identifier that is different from the identifier of the existing data stream.
13. The method of claim 11 , further comprising inheriting, by the first data stream, one or more application programs registered to the existing data stream.
14. The method of claim 11 , further comprising registering an application to the second data stream.
15. The method of claim 11 , further comprising closing an epoch of the existing data stream prior to creating the new first stream fragment.
16. The method of claim 11 , further comprising closing an epoch of the existing data stream after creating the new first stream fragment.
17. The method of claim 11 , further comprising receiving a truncation request directed to truncating older data of the first data stream, and in response to the truncation request, moving a head location of the first data stream to a new head location further forward in the first data stream, and determining whether at least one data stream has any head location prior to the new head location, and
in response to determining that no data stream has any head location prior to the new head location, deleting the older data, and
in response to determining that at least one data stream has at least one head location prior to the new head location, maintaining the older data for the at least one data stream.
18. A non-transitory machine-readable medium, comprising executable instructions that, when executed by a processor of a streaming data storage system, facilitate performance of operations, the operations comprising:
appending data events obtained from a writer application program to an existing data stream;
receiving a request to fork the existing data stream;
in response to receiving the request, creating a first new stream fragment at a tail of the existing data stream, and creating a second new stream fragment at the tail of the existing data stream;
receiving a new data event from the writer; and
in response to the receiving the new data event, appending the new data event to the first new stream fragment without appending the new data event to the second new stream fragment.
19. The non-transitory machine-readable medium of claim 18 , wherein the writer is a first writer, wherein the new data event is a first data event, and wherein the operations further comprise receiving, from a second writer, a second data event directed to the second new stream fragment, and in response to the receiving the second data event, appending the second data event to the second new stream fragment without appending the second data event to the first new stream fragment.
20. The non-transitory machine-readable medium of claim 18 , wherein the operations further comprise,
receiving a truncation request directed to truncating older data of the first data stream,
in response to receiving the truncation request, moving a head location of the first data stream to a new head location further forward in the first data stream, and determining whether at least one data stream has any head location prior to the new head location,
in response to determining that no data stream has any head location prior to the new head location, deleting the older data, and
in response to determining that at least one data stream has at least one head location prior to the new head location, maintaining the older data for the at least one data stream.