Systems and methods for consistent backup of distributed, transactional databases
A distributed, transactional database uses timestamps, such as logical clock values, for entry versioning and transaction management in the database. To write to the database, a service requests a timestamp to be inserted into the database with a new version of data. During a backup procedure, a cleanup process is paused, issuing new timestamps is paused, and a backup timestamp is generated, which results in an effective backup copy. During a restore of a backup, a snapshot of the database is loaded and any entries older than the backup timestamp are deleted, which ensures that a consistent restore has occurred.
1. A distributed database system comprising:
one or more non-transitory computer storage mediums configured to store one or more database clusters, the one or more database clusters comprising a plurality of nodes, wherein each node of the plurality of nodes comprises a key-value table, and wherein each entry of a respective key-value table comprises a timestamp;
a backup data store; and
one or more computer hardware processors in communication with the one or more non-transitory computer storage mediums, the one or more computer hardware processors configured to execute computer-executable instructions to at least:
pause a cleanup process on the plurality of nodes;
initiate a hold on issuing one or more new timestamps to one or more external computing devices;
generate a backup timestamp;
remove the hold on issuing one or more new timestamps to one or more external computing devices;
generate a duplicate copy of data from the plurality of nodes;
initiate the cleanup process on the plurality of nodes; and
store, in the backup data store, the duplicate copy of data and the backup timestamp.
2. The system of claim 1 , wherein the one or more computer hardware processors are further configured to:
receive a request to restore one or more key-value tables on the plurality of nodes;
load the duplicate copy of data on a first node of the plurality of nodes; and
delete a first entry from the first node with a first timestamp more recent than the backup timestamp.
3. The system of claim 2 , wherein to store the duplicate copy of data, the one or more computer hardware processors are further configured to:
determine that a first portion of the duplicate copy of data changed from a previous duplicate copy of data; and
store, in the backup data store, the first portion of the duplicate copy of data that changed.
4. The system of claim 3 , wherein to store the duplicate copy of data, the one or more computer hardware processors are further configured to:
determine that a second portion of the duplicate copy of data remained unchanged from the previous duplicate copy of data; and
store, in the backup data store, metadata associated with the second portion, the metadata indicating that the second portion remained unchanged.
5. The system of claim 3 , wherein to determine that the first portion changed, the one or more computer hardware processors are further configured to:
generate a first hash value from the first portion;
generate a second hash value from a portion of the previous duplicate copy of data corresponding to the first portion; and
determine that the first hash value is different than the second hash value.
6. The system of claim 1 , wherein the duplicate copy of data comprises a portion of data from the plurality of nodes.
7. The system of claim 6 , wherein the portion of data comprises data that has changed since a last backup.
8. A method comprising:
pausing a cleanup process on a plurality of nodes on one or more database clusters, wherein each node of the plurality of nodes comprises a key-value table, and wherein each entry of a respective key-value table comprises a timestamp;
initiating a hold on allowing writes from committing to the one or more database clusters;
generating a backup timestamp;
removing the hold on allowing writes from committing to the one or more database clusters;
generating a duplicate copy of data from the plurality of nodes;
initiating the cleanup process on the plurality of nodes; and
storing, in a backup data store, the duplicate copy of data and the backup timestamp,
wherein the method is performed by one or more computer hardware processors.
9. The method of claim 8 , further comprising:
receiving a request to restore one or more key-value tables on the plurality of nodes;
loading the duplicate copy of data on a first node of the plurality of nodes; and
deleting a first entry from the first node with a first timestamp more recent than the backup timestamp.
10. The method of claim 9 , wherein the first timestamp comprises a first numerical value, the backup timestamp comprises a second numerical value, the method further comprising:
determining that the first timestamp is more recent than the backup timestamp, wherein determining that the first timestamp is more recent further comprises:
determining that the first numerical value is greater than the second numerical value.
11. The method of claim 8 , wherein the backup timestamp comprises a logical clock value, and wherein generating the backup timestamp further comprises incrementing a current logical clock value.
12. The method of claim 8 , further comprising:
receiving a request to insert a key-value entry into a first key-value table on the plurality of nodes;
generating a new timestamp; and
inserting the key-value entry into the first key-value table with the new timestamp.
13. The method of claim 12 , wherein the new timestamp comprises a logical clock value, and wherein generating the new timestamp further comprises incrementing a current logical clock value.
14. The method of claim 8 , further comprising:
receiving, from a first external computing device, a request for a new timestamp during the hold; and
postponing responding to the request until the hold is removed.