Database recovery time objective optimization with synthetic snapshots
Methods and systems for reducing the amount of time to restore a database or other application by dynamically generating and storing synthetic snapshots are described. When backing up a database, an integrated data management and storage system may acquire snapshots of the database at a snapshot frequency and acquire database transaction logs at a frequency that is greater than the snapshot frequency. In response to detecting that the database is unable to provide a database snapshot, the integrated data management and storage system may generate a synthetic snapshot of the database by instantiating a compatible version of the database locally, acquiring a previously stored snapshot of the database, applying data changes from one or more database transaction logs to the previously stored snapshot to generate the synthetic snapshot, and storing the synthetic snapshot of the database within the integrated data management and storage system.
1. A method for operating a data management system at a storage appliance, comprising:
determining, by the storage appliance, a snapshot frequency for capturing snapshots of a database from a server, wherein the server is separate from the storage appliance;
acquiring, at the storage appliance, a first snapshot of a database at a first point in time from a server executing the database;
determining, by the storage appliance, that a quantity of one or more transaction logs for the database acquired between the first point in time and a second point in time exceeds a pre-determined threshold value, wherein the one or more transaction logs for the database include a set of data changes to the database between the first point in time and the second point in time subsequent to the first point in time, and wherein the one or more transaction logs are acquired from the server at a frequency that is higher than the snapshot frequency for the database;
requesting, by the storage appliance based at least in part on determining that the quantity of one or more transaction logs acquired between the first point in time and the second point in time exceeds the pre-determined threshold value, that the server provide a second snapshot of the database at the second point in time;
detecting, by the storage appliance, that the server is unable to provide the second snapshot of the database to the storage appliance within a threshold time period that begins at a time that the server is requested to provide the second snapshot;
determining, by the storage appliance, that a synthetic snapshot of the database is to be generated based at least in part on determining that the quantity of one or more transaction logs for the database exceeds a pre-determined threshold value and based at least in part on detecting that the server is unable to provide the second snapshot of the database within the threshold time period;
instantiating, by the storage appliance, a database engine to generate the synthetic snapshot for the database;
generating, by the storage appliance, the synthetic snapshot of the database by applying the set of data changes to the first snapshot of the database using the database engine;
terminating, by the storage appliance, the database engine after generating the synthetic snapshot using the database engine; and
storing, within the storage appliance, the synthetic snapshot of the database.
2. The method of claim 1 , wherein detecting that the server is unable to provide the second snapshot of the database includes determining a blackout window for the database in which snapshot acquisition from the database is prohibited and detecting that the second point in time is during the blackout window for the database.
3. The method of claim 1 , further comprising:
determining a compatibility level for the database; and
determining the database engine based at least in part on the compatibility level for the database.
4. The method of claim 1 , further comprising:
detecting that the server is unable to provide a third snapshot of the database at a third point in time subsequent to the second point in time;
acquiring one or more additional transaction logs for the database that include a second set of data changes to the database between the second point in time and the third point in time; and
generating the third snapshot of the database using the second set of data changes and the synthetic snapshot of the database using the database engine.
5. The method of claim 1 , wherein the one or more transaction logs for the database comprise a plurality of database transaction logs.
6. The method of claim 1 , wherein storing the synthetic snapshot of the database includes storing the synthetic snapshot as an incremental snapshot.
7. The method of claim 1 , wherein the database comprises a relational database.
8. The method of claim 1 , further comprising:
detecting that a quantity of updates to the database between the first point in time and the second point in time has exceeded a threshold quantity of updates; and
outputting an alert specifying that a database anomaly has been detected.
9. A data management system at a storage appliance, comprising:
a memory storing instructions; and
one or more hardware processors communicatively coupled to the memory and configured by the instructions to perform operations comprising:
determining, by the storage appliance, a snapshot frequency for capturing snapshots of a database from a server, wherein the server is separate from the storage appliance;
acquiring, at the storage appliance, a first snapshot of a database at a first point in time from a server executing the database;
determining, by the storage appliance, that a quantity of one or more transaction logs for the database acquired between the first point in time and a second point in time exceeds a pre-determined threshold value, wherein the one or more transaction logs for the database include a set of data changes to the database between the first point in time and the second point in time subsequent to the first point in time, and wherein the one or more transaction logs are acquired from the server at a frequency that is higher than the snapshot frequency for the database;
requesting, by the storage appliance based at least in part on determining that the quantity of one or more transaction logs acquired between the first point in time and the second point in time exceeds the pre-determined threshold value, that the server provide a second snapshot of the database at the second point in time;
detecting, by the storage appliance, that the server is unable to provide the second snapshot of the database to the storage appliance within a threshold time period that begins at a time that the server is requested to provide the second snapshot;
determining, by the storage appliance, that a synthetic snapshot of the database is to be generated based at least in part on determining that the quantity of one or more transaction logs for the database exceeds a pre-determined threshold value and based at least in part on detecting that the server is unable to provide the second snapshot of the database within the threshold time period;
instantiating, by the storage appliance, a database engine to generate the synthetic snapshot for the database;
generating, by the storage appliance, the second synthetic snapshot of the database by applying of the set of data changes to the first snapshot of the database using the database engine;
terminating, by the storage appliance, the database engine after generating the synthetic snapshot using the database engine; and
storing, within the storage appliance, the synthetic snapshot of the database.
10. The data management system of claim 9 , wherein the operations comprise:
determining a blackout window for the database in which snapshot acquisition from the database is prohibited and detect that the second point in time is during the blackout window for the database.
11. The data management system of claim 9 , wherein the operations comprise:
determining a compatibility level for the database and determining the database engine based at least in part on the compatibility level for the database.
12. The data management system of claim 9 , wherein the operations comprise:
detecting that the server is unable to provide a third snapshot of the database at a third point in time subsequent to the second point in time and acquiring one or more additional transaction logs for the database that include a second set of data changes to the database between the second point in time and the third point in time; and
using the database engine to generate the third snapshot of the database based at least in part on the second set of data changes and the synthetic snapshot of the database.
13. The data management system of claim 9 , wherein the one or more transaction logs for the database comprise a plurality of database transaction logs.
14. The data management system of claim 9 , wherein the synthetic snapshot is stored as a full image snapshot.
15. The data management system of claim 9 , wherein the database comprises a non-relational database.
16. A method for operating a data management system at a storage appliance, comprising:
determining, by the storage appliance, a snapshot frequency for capturing snapshots of a database from a server, wherein the server is separate from the storage appliance;
acquiring, at the storage appliance, a first snapshot of the database at a first point in time from the server executing the database;
determining, by the storage appliance, that a quantity of one or more transaction logs for the database acquired between the first point in time and a second point in time exceeds a pre-determined threshold value, wherein the one or more transaction logs for the database include a set of data changes that have occurred to the database between the first point in time and the second point in time subsequent to the first point in time, and wherein the one or more transaction logs are acquired from the server at a frequency that is higher than the snapshot frequency for the database;
requesting, by the storage appliance based at least in part on determining that the quantity of one or more transaction logs acquired between the first point in time and the second point in time exceeds the pre-determined threshold value, that the server provide a second snapshot of the database at the second point in time;
detecting, by the storage appliance, that the server is unable to provide the second snapshot of the database to the storage appliance within a threshold time period that begins at a time that the server is requested to provide the second snapshot and that an amount of time between the first point in time and the second point in time is less than a second threshold time period determined based at least in part on the snapshot frequency;
determining, by the storage appliance, that a synthetic snapshot of the database is to be generated based at least in part on determining that the quantity of one or more transaction logs for the database exceeds a pre-determined threshold value and based at least in part on the detecting;
identifying, by the storage appliance, a database engine to generate the synthetic snapshot for the database;
generating, by the storage appliance, the synthetic snapshot by applying the set of data changes to the database using the database engine;
terminating, by the storage appliance, the database engine after generating the second snapshot using the database engine; and
storing, within the storage appliance, the synthetic snapshot of the database.
17. The method of claim 16 , wherein determining that the quantity of one or more transaction logs for the database acquired between the first point in time and the second point in time exceeds the pre-determined threshold value includes detecting that an aggregate file size for the one or more transaction logs is greater than a threshold file size.