Live data conversion and migration for distributed data object systems
A system converts the data of data objects stored in a DDOS from one data format to another while the system is live and available to process requests for access to the data objects being converted. This process does not require taking the DDOS offline and also does not require locking a data object for the entire conversion of the data object.
1. A computing system comprising:
a non-transitory computer-readable storage medium comprising instructions encoded therein; and
one or more processors that, when executing the instructions, cause the system to perform operations comprising:
storing a first set of database tables based on a second set of database tables;
obtaining a first set of one or more signatures for a target database table of the first set of database tables; and
for a source database table of the second set of database tables:
acquiring a lock on the source database table,
obtaining a second set of one or more signatures for the source database table,
determining a set of one or more mismatched database table rows between the target database table and the source database table based on a comparison of the first set of one or more signatures to the second set of one or more signatures,
determining whether a number of mismatched rows in the set of one or more mismatched database table rows does not exceed a threshold number of mismatched rows, and
in response to determining that the number of mismatched rows in the set of one or more mismatched database table rows does not exceed the threshold number of mismatched rows, releasing the lock on the source database table after copying the set of one or more mismatched database table rows from the source database table to the target database table.
2. The computing system of claim 1 , wherein the source database table and the target database table store data using different formats with respect to one another.
3. The computing system of claim 1 , the operations further comprising, for the source database table in the second set of database tables:
in response to determining that the number of mismatched rows in the set of one or more mismatched database table rows exceeds the threshold number of mismatched rows, releasing the lock on the source database table before performing the copying the set of one or more mismatched database table rows from the source database table to the target database table.
4. The computing system of claim 1 , the operations further comprising obtaining the first set of one or more signatures from a metadata store.
5. The computing system of claim 1 , the operations further comprising computing the first set of one or more signatures from the target database table.
6. The computing system of claim 1 , wherein the source database table uses a record columnar format and wherein the target database table uses an optimized row columnar format.
7. The computing system of claim 1 , wherein the lock is a write lock, and wherein the lock acquired on the source database table does not prevent reading of the source database table.
8. A computer-implemented method comprising:
storing a first set of database tables based on a second set of database tables;
obtaining a first set of one or more signatures for a target database table of the first set of database tables; and
for a source database table of the second set of database tables:
acquiring a lock on the source database table,
obtaining a second set of one or more signatures for the source database table,
determining a set of one or more mismatched database table rows between the target database table and the source database table based on a comparison of the first set of one or more signatures to the second set of one or more signatures,
determining whether a number of mismatched rows in the set of one or more mismatched database table rows does not exceed a threshold number of mismatched rows, and
in response to determining that the number of mismatched rows in the set of one or more mismatched database table rows does not exceed the threshold number of mismatched rows, releasing the lock on the source database table after copying the set of one or more mismatched database table rows from the source database table to the target database table.
9. The computer-implemented method of claim 8 , wherein the source database table and the target database table store data using different formats with respect to one another.
10. The computer-implemented method of claim 8 , further comprising, for the source database table in the second set of database tables:
in response to determining that the number of mismatched rows in the set of one or more mismatched database table rows exceeds the threshold number of mismatched rows, releasing the lock on the source database table before performing the copying the set of one or more mismatched database table rows from the source database table to the target database table.
11. The computer-implemented method of claim 8 , further comprising obtaining the first set of one or more signatures from a metadata store.
12. The computer-implemented method of claim 8 , further comprising computing the first set of one or more signatures from the target database table.
13. The computer-implemented method of claim 8 , wherein the source database table uses a record columnar format and the target database table uses an optimized row columnar format.
14. The computer-implemented method of claim 8 , wherein the lock is a write lock, and wherein the lock acquired on the source database table does not prevent reading of the source database table.
15. A non-transitory computer-readable medium comprising instructions encoded thereon that, when executed, cause one or more processors to perform operations comprising:
storing a first set of database tables based on a second set of database tables;
obtaining a first set of one or more signatures for a target database table of the first set of database tables; and
for a source database table of the second set of database tables:
acquiring a lock on the source database table,
obtaining a second set of one or more signatures for the source database table,
determining a set of one or more mismatched database table rows between the target database table and the source database table based on a comparison of the first set of one or more signatures to the second set of one or more signatures,
determining whether a number of mismatched rows in the set of one or more mismatched database table rows does not exceed a threshold number of mismatched rows, and
in response to determining that the number of mismatched rows in the set of one or more mismatched database table rows does not exceed the threshold number of mismatched rows, releasing the lock on the source database table after copying the set of one or more mismatched database table rows from the source database table to the target database table.
16. The non-transitory computer-readable medium of claim 15 , wherein the source database table and the target database table store data using different formats with respect to one another.
17. The non-transitory computer-readable medium of claim 15 , the operations further comprising, for the source database table in the second set of database tables:
in response to determining that the number of mismatched rows in the set of one or more mismatched database table rows exceeds the threshold number of mismatched rows, releasing the lock on the source database table before performing the copying the set of one or more mismatched database table rows from the source database table to the target database table.
18. The non-transitory computer-readable medium of claim 15 , the operations further comprising obtaining the first set of one or more signatures from a metadata store.
19. The non-transitory computer-readable medium of claim 15 , the operations further comprising computing the first set of one or more signatures from the target database table.
20. The non-transitory computer-readable medium of claim 15 , wherein the source data format is a record columnar format and the target data format is an optimized row columnar format.