Fast database recovery in a multi-volume database environment via transactional awareness
Techniques for fast database recovery in a multi-volume database environment via transactional awareness are described. In the event of a failure associated with a first volume storing database page data, the first volume can be restored to a point in time and transactional metadata from a second volume storing logical change data can be obtained for a limited number of transactions occurring at/after that point in time, as opposed to analyzing extremely large change log files. These transactions can be checked to ensure that they have all been persisted, and if not, change data for those transactions can be obtained from the second volume and used to replay these transactions on the restored first volume.
1. A computer-implemented method comprising:
detecting a failure associated with a database implemented by a database service in a multi-tenant cloud provider network, the database service including a first electronic device implementing a table volume that stores and/or indexes logical change data of the database, the logical change data pertaining to committed updates that occurred between a time of the failure and a restored time, and a second electronic device implementing a page volume that stores page change data for database pages of the database used by a database engine implemented by a third electronic device of the database service;
determining to recover the database;
obtaining, by a control plane of the database service via use of a first data structure in the table volume of the first electronic device, transactional metadata for one or more transactions that were committed into a last file of multiple files storing the logical change data of the database in a second data structure of the table volume of the first electronic device;
verifying whether data for the one or more transactions was durably written to the page volume implemented by the second electronic device;
obtaining the logical change data, from the second data structure of the table volume of the first electronic device, for one of the one or more transactions determined to not have been durably written to the page volume of the second electronic device; and
sending one or more commands to the database engine, based at least in part on the logical change data, to cause the database engine to replay at least the one transaction to update the page volume.
2. The computer-implemented method of claim 1 , wherein the logical change data stored in the second data structure comprises changelog records, and wherein the page change data stored by the page volume includes redo log records.
3. The computer-implemented method of claim 1 , wherein the obtaining of the transactional metadata includes collecting transactional metadata for only those transactions, of the transactions committed into the last file, that are not marked as being durably committed within the first data structure of the table volume, and wherein the first data structure comprises a transaction table.
4. A computer-implemented method performed by a database service including a first electronic device implementing a table volume that stores and/or indexes logical change data of a database, and a second electronic device implementing a page volume that stores page change data for database pages of the database used by a database engine implemented by a third electronic device of the database service, the method comprising:
detecting a failure associated with the database;
obtaining, via the table volume of the database implemented by the first electronic device, transactional metadata for one or more transactions that were committed into a last file of multiple files storing the logical change data of the database, the logical change data pertaining to committed updates that occurred between a time of the failure and a restored time;
verifying whether data for the one or more transactions was durably written to the page volume implemented by the second electronic device;
obtaining the logical change data, from the table volume of the first electronic device, for one of the one or more transactions determined to not have been durably written to the page volume of the second electronic device; and
sending one or more commands, based at least in part on the logical change data, to cause at least the one transaction to be replayed to update the page volume of the second electronic device.
5. The computer-implemented method of claim 4 , wherein the obtaining of the transactional metadata includes collecting transactional metadata for all transactions that were committed into the last file.
6. The computer-implemented method of claim 4 , wherein the obtaining of the transactional metadata includes collecting transactional metadata for only those transactions, of the transactions committed into the last file, that are not marked as being durably committed within a transaction table of the table volume.
7. The computer-implemented method of claim 4 , wherein:
the obtained transactional metadata does not include actual logical change data associated with the one or more transactions;
the obtained transactional metadata is stored in a first data structure of the table volume; and
the actual logical change data associated with the one or more transactions is stored in a second data structure of the table volume.
8. The computer-implemented method of claim 4 , wherein the obtaining of the transactional metadata, the verifying, the obtaining of the logical change data, and the replay of at least the one transaction to update the page volume occurs in an amount of time, the amount of time being less than ten seconds.
9. The computer-implemented method of claim 8 , wherein the amount of time is less than one second.
10. The computer-implemented method of claim 4 , wherein the database is implemented by the database service in a multi-tenant cloud provider network, and wherein the database service is communicatively coupled to the table volume and the database page volume via one or more network connections.
11. The computer-implemented method of claim 4 , wherein the page change data stored by the page volume includes redo log records, and wherein the page volume further stores database pages.
12. The computer-implemented method of claim 11 , wherein the logical change data stored by the table volume includes changelog records.
13. The computer-implemented method of claim 4 , wherein a size of the obtained transactional metadata is less than ten percent of a size of a corresponding one of the multiple files, and wherein the multiple files storing the logical change data of the database are binlog files.
14. The computer-implemented method of claim 4 , wherein the transactional metadata, for at least one transaction of the one or more transactions, includes a commit time of the at least one transaction, a transaction identifier of the at least one transaction, or a log sequence number (LSN) associated with the table volume.
15. A system implemented as code executing on one or more processors, the system comprising:
a first one or more electronic devices to implement a table volume that stores and/or indexes logical change data of a database of a database service, the logical change data pertaining to committed updates that occurred between a time of the failure and a restored time;
a second one or more electronic devices to implement a page volume that stores page change data for database pages of the database;
a third one or more electronic devices to implement a database engine of the database service, the database engine to utilize at least the page volume and the table volume; and
a fourth one or more electronic devices to implement a control plane of the database service, the control plane including instructions that upon execution cause the control plane to:
detect a failure associated with the database;
obtain, via the table volume of the database implemented by the first electronic device, transactional metadata for one or more transactions that were committed into a last file of multiple files storing the logical change data of the database, the logical change data pertaining to committed updates that occurred between a time of the failure and a restored time;
verify whether data for the one or more transactions was durably written to the page volume implemented by the second electronic device;
obtain the logical change data, from the table volume of the first electronic device, for one of the one or more transactions determined to not have been durably written to the page volume of the second electronic device; and
send one or more commands, based at least in part on the logical change data, to cause at least the one transaction to be replayed to update the page volume of the second electronic device.
16. The system of claim 15 , wherein the obtaining of the transactional metadata includes collecting transactional metadata for all transactions that were committed into the last file.
17. The system of claim 15 , wherein the obtaining of the transactional metadata includes collecting transactional metadata for only transactions, of the transactions committed into the last file, that are not marked as being durably committed within a transaction table of the table volume.
18. The system of claim 15 , wherein:
the obtained transactional metadata does not include actual logical change data associated with the one or more transactions;
the obtained transactional metadata is stored in a first data structure of the table volume; and
the actual logical change data associated with the one or more transactions is stored in a second data structure of the table volume.
19. The system of claim 15 , wherein the obtaining of the transactional metadata, the verification, the obtaining the logical change data, and the replay, all occur in an amount of time, the amount of time being less than ten seconds.
20. The system of claim 15 , wherein the page change data stored by the page volume includes redo log records, wherein the page volume further stores database pages, and wherein the logical change data stored by the table volume includes changelog records.