Data transfer resiliency during bulk to streaming transition
An indication to migrate requested data objects stored in a source database environment to a destination database environment is received. Some of data objects have many-to-one relationships with other data objects in the source database environment. At least one snapshot file generated by the source database environment is transferred to a destination database environment in bulk transfer mode. Subsequent incoming data received during bulk transfer mode, after the indication is stored in a temporary table. Upon completion of migration of the requested data objects the system transitions from bulk transfer mode to streaming mode. The subsequent incoming data from the temporary table is transferred to the destination database environment in response to the transition to streaming mode. Additional data received after the temporary table is empty is transferred from the source database environment to the destination database environment without use of the temporary table.
1. A system comprising:
a processor;
memory;
a first database environment to store source data having objects in a first format having at least some of objects having many-to-one relationships with one or more other objects;
a second database environment to receive data objects from the first database environment through a streaming mechanism supporting multiple streaming topics with corresponding data retention periods; and
a data manager agent coupled with the first database environment and with the second database environment to facilitate a migration process via a bulk data transfer, wherein facilitating the migration process includes determining whether the time it takes to complete the migration process would exceed a predetermined data retention window threshold and when the time it takes to complete the migration process would exceed the predetermined data retention window, then the data manager agent will write data to a temporary database table to be stored during the migration process prior to transmission to selected streaming topics in the second database environment, and, upon completion of the migration process, perform a state transition from a bulk mode to a streaming mode by ingesting data from the temporary database using a streaming data transfer before ingesting data from the first database environment, when the time it takes to complete the migration process would not exceed the predetermined data retention window, new data received in a streaming mode can be read starting at an offset value recorded at the start of the migration process.
2. The system of claim 1 wherein the temporary database table is stored in the second database environment.
3. The system of claim 2 wherein a size of the temporary database table is dynamically modifiable.
4. The system of claim 1 wherein the temporary database table stores data for a designated streaming topic from the selected streaming topics.
5. The system of claim 1 wherein the temporary database table provides a data storage period that is greater than the data retention period of the corresponding stream.
6. The system of claim 1 wherein the first database environment generates one or more snapshot files to provide copies of selected objects at a corresponding point in time.
7. The system of claim 6 wherein the migration process comprises at least migrating one or more snapshot files to the second database environment.
8. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, are configurable to cause the one or more processors to:
detect an indication to migrate requested data stored in a source database environment to a destination database environment, wherein the source database environment generates one or more snapshot files to provide copies of data objects at a corresponding point in time;
transfer at least one snapshot file to a destination database environment as part of a migration process in a bulk transfer mode;
determine whether the time it takes to complete the migration process would exceed a predetermined data retention window threshold;
when it is determined that the time it takes to complete the migration process would exceed the predetermined data retention window, then:
store subsequent incoming data received during the bulk transfer mode and after the indication in a temporary database table, if any;
transition from the bulk transfer mode to a streaming mode in response to completion of migration of the requested data; and
transfer the subsequent incoming data, if any, from the temporary database table to the destination database environment using a streaming data transfer in response to the transition to the streaming mode; and
when it is determined that the time it takes to complete the migration process would not exceed the predetermined data retention window, then transfer incoming data received after the migration process has completed to the destination database environment starting at an offset value recorded at the start of the migration process.
9. The non-transitory computer-readable medium of claim 8 wherein the temporary database table is stored in the second database environment.
10. The non-transitory computer-readable medium of claim 9 wherein a size of the temporary database table is dynamically modifiable.
11. The non-transitory computer-readable medium of claim 8 wherein the temporary database table stores data for a designated streaming topic from multiple streaming topics supported by the source database environment.
12. The non-transitory computer-readable medium of claim 11 wherein the temporary database table provides a data storage period that is greater than a data retention period of the corresponding stream.
13. The non-transitory computer-readable medium of claim 8 wherein the source database environment stores database objects in a first format and at least some of objects have many-to-one relationships with one or more other objects in the source database environment.
14. The non-transitory computer-readable medium of claim 8 wherein the one or more snapshot files are stored in a database external to the source database environment.
15. A method comprising:
receiving an indication to migrate requested data objects stored in a source database environment to a destination database environment, wherein at least some of data objects have many-to-one relationships with one or more other data objects in the source database environment;
transfer at least one snapshot file to a destination database environment as part of a migration process in a bulk transfer mode;
determine whether the time it takes to complete the migration process would exceed a predetermined data retention window threshold;
when it is determined that the time it takes to complete the migration process would exceed the predetermined data retention window, then:
store subsequent incoming data received during the bulk transfer mode and after the indication in a temporary database table, if any;
transition from the bulk transfer mode to a streaming mode in response to completion of migration of the requested data; and
transfer the subsequent incoming data, if any, from the temporary database table to the destination database environment using a streaming data transfer in response to the transition to the streaming mode; and
when it is determined that the time it takes to complete the migration process would not exceed the predetermined data retention window, then transfer incoming data received after the migration process has completed to the destination database environment starting at an offset value recorded at the start of the migration process.
16. The method of claim 15 wherein the temporary database table is stored in the second database environment.
17. The method of claim 15 wherein a size of the temporary database table is dynamically modifiable.
18. The method of claim 15 wherein the temporary database table stores data for a designated streaming topic from multiple streaming topics supported by the source database environment.
19. The method of claim 15 wherein the temporary database table provides a data storage period that is greater than a data retention period of the corresponding stream.
20. The method of claim 15 wherein the one or more snapshot files are stored in a database external to the source database environment.