Migration of legacy data into an ordered event stream
Data migration from a legacy data storage system into an ordered event stream (OES) data storage system is disclosed. In contrast to conventional techniques, the disclosed subject matter can provide for migrating legacy data into a first portion of a vintage OES (VOES) and new data, intended for the legacy data storage system during the migration process, into a second portion of the VOES. The first and second portions of the VOES can then be linked at completion of the migration process. Avoiding writing new data to the legacy data storage system prior to migration enables a corresponding reduction in data access disruption. Moreover, event sequencing by separating migrated legacy events from new events can preserving event order. Scaling of vintage streams can be supported. Vintage streams can support function calls to a single VOES in contrast to the more conventional use of multiple streams.
1. A system, comprising:
a processor; and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, comprising:
migrating legacy data, as a legacy event, to a first portion of an ordered event stream during a legacy data migration, wherein the legacy data is stored via a legacy data storage system, wherein migrating the legacy data comprises determining an event order corresponding to writing the legacy event to the first portion of the ordered event stream in accord with a determined event sequence, wherein the event order is determined based on analysis of the legacy data due to a lack of data ordering information being provided by the legacy data storage system, and wherein the legacy data storage system is an unstructured data storage system;
writing a new event to a second portion of the ordered event stream, wherein the new event is based on new data received during the legacy data migration and wherein the new event is not yet stored via the legacy data storage system; and
in response to the legacy data migration being determined to have concluded, linking between a last event of the first portion of the ordered event stream and a first event of the second portion of the ordered event stream, resulting in the ordered event stream comprising both the legacy event and the new event, each in accord with the determined event sequence.
2. The system of claim 1 , wherein writing the new event to the second portion of the ordered event stream is in lieu of writing the new data to the legacy data storage system.
3. The system of claim 1 , wherein the ordered event stream is generated faster than another ordered event stream, wherein the other ordered event stream is generated by migrating the legacy data to the other ordered event stream, writing the new data to the legacy data storage system, and further migrating the new data from the legacy data storage system to the other ordered event stream.
4. The system of claim 1 , wherein the event order is determined based on a legacy data order corresponding to storing the legacy data via the legacy data storage system.
5. The system of claim 1 , wherein the event order is determined based on data ordering information comprised in the legacy data.
6. The system of claim 1 , wherein the event order is determined based on data ordering information determined from an analysis of the legacy data rather than on the data ordering information comprised in the legacy data.
7. The system of claim 1 , wherein events comprising the legacy event and the new event are readable from the ordered event stream in a sequence that accords with the determined event sequence.
8. The system of claim 1 , wherein the ordered event stream supports scaling operations subsequent to the legacy data migration being determined to have concluded.
9. The system of claim 1 , wherein the ordered event stream enables function calls to a single namespace in relation to both the legacy event and the new event.
10. A method, comprising:
in response to beginning a migration of legacy data from a legacy data storage system to an ordered event stream data storage system, migrating, by a system comprising a processor, legacy data as a legacy event to a first portion of an ordered event stream of the ordered event stream data storage system, wherein the migrating comprises determining an event order corresponding to writing the legacy event to the first portion of the ordered event stream in accord with a determined event sequence, and wherein based on data ordering information corresponding to the legacy event is not received from the legacy data storage system, the event order is determined based on an analysis of the legacy data;
writing, by the system, a new event to a second portion of the ordered event stream, wherein the new event corresponds to new data targeted for storage via the legacy data storage system, and wherein writing the new event to the second portion of the ordered event stream supersedes writing the new data to the legacy data storage system; and
generating, by the system, a logical connection from a last event of the first portion of the ordered event stream to a first event of the second portion of the ordered event stream in response to determining completion of the migration, wherein the logical connected enables sequential reading of events from the first portion of the ordered event stream to the second portion of the ordered event stream.
11. The method of claim 10 , wherein the ordered event stream data storage system is a first ordered event stream data storage system, and wherein the legacy data storage system is not an ordered event stream data storage system.
12. The method of claim 10 , wherein the legacy data storage system is an unstructured data storage system that is not an ordered event stream data storage system.
13. The method of claim 10 , wherein the event order is determined based on a legacy data order corresponding to storing the legacy data via the legacy data storage system.
14. The method of claim 10 , wherein the event order is determined based on data ordering information determined from an analysis of the legacy data rather than on the data ordering information comprised in the legacy data.
15. The method of claim 10 , wherein the event order is determined based on data ordering information comprised in the legacy data.
16. A non-transitory machine-readable medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising:
generating a legacy event in a first portion of an ordered event stream during a legacy data migration, wherein generating the legacy event is based on legacy data stored via a legacy data storage system, wherein the legacy data migration comprises determining an event order corresponding to the generating the legacy event in the first portion of the ordered event stream in accord with a determined event sequence, and wherein, based on data ordering information corresponding to the legacy event not being received from the legacy data storage system, the event order is determined based on an analysis of the legacy data;
generating a new event in a second portion of the ordered event stream during the legacy data migration, wherein generating the new event is based on new data, and wherein generating the new event is performed in lieu of writing the new data to the legacy data storage system; and
in response to determining the legacy data migration has ended, connecting a last event of the first portion of the ordered event stream to a first event of the second portion of the ordered event stream, resulting in a contiguous ordered event stream comprising the legacy event, in accord with the determined event sequence, and the new event, also in accord with the determined event sequence.
17. The non-transitory machine-readable medium of claim 16 , wherein the determined event sequence is determined from an analysis of the legacy data.
18. The non-transitory machine-readable medium of claim 16 , wherein the determined event sequence is determined from sequence information comprised in the legacy data.
19. The non-transitory machine-readable medium of claim 16 , wherein the legacy data storage system is an unstructured data storage system that is not an ordered event stream data storage system.
20. The non-transitory machine-readable medium of claim 16 , wherein the event order is determined based on data ordering information determined from an analysis of the legacy data rather than on the data ordering information comprised in the legacy data.