Backup and tiered policy coordination in time series databases
A data protection system configured to backup a time series database is provided. The data protection system may be integrated with or have access to consolidation policies of the time series database. The backup policy and backup retention policy are set by monitoring the consolidation policy and adjusting the backup policy to ensure that the data in the time series database is protected prior to being downscaled, discarded or otherwise consolidated.
1. A computer-implemented method for backing up a time series database, the method comprising:
monitoring, by a data protection system that includes one or more processors, a consolidation policy of the time series database, wherein consolidation policy includes consolidation data that identifies when data in the time series database is downscaled, the data including time series data, wherein the data in the time series database loses resolution when the data is downscaled;
defining, by the one or more processors, a backup policy of the data protection system based on the consolidation policy of the time series database;
generating, by the one or more processors, backups of the data in the time series database in accordance with the backup policy by the data protection system, wherein the backup policy ensures that the backups are generated before the data in the time series database is downscaled or discarded such that the backups contain full resolution copies of the time series data, wherein the backups further include backups of downscaled versions of the data in the time series database, wherein the downscaled versions have different resolutions of the data in the time series database; and
updating, by the one or more processors, a catalog that identifies where the backups are stored and resolutions of the backups.
2. The method of claim 1 , further comprising tracking changes to the consolidation policy, wherein the consolidation policy is read continuously, periodically, or when instructed.
3. The method of claim 1 , further comprising detecting, by the one or more processors, a change in the consolidation policy and dynamically adjusting the backup policy whenever the consolidation policy changes in order to accommodate the detected change to the consolidation policy.
4. The method of claim 1 , wherein different data sets in the time series data have different consolidation policies.
5. The method of claim 1 , further comprising monitoring usage of the backups and adjusting a retention period of the backups based on the monitored usage, wherein the retention period is based on user preferences and wherein the user preferences express a boundary for the retention period.
6. The method of claim 1 , further comprising adjusting a backup frequency when a time that the time series data is kept at a high resolution changes.
7. The method of claim 1 , further comprising creating a catalog to include in the backups, wherein the catalog identifies locations of the data and a resolution of the data.
8. The method of claim 1 , further comprising ensuring that a full resolution of the data is backed up before older data is downscaled.
9. The method of claim 1 , further comprising retraining a machine learning model based on older data that is at full resolution or older data that is at less than full resolution.
10. The method of claim 1 , wherein the backups include full resolutions of the data stored in the time series database, further comprising archiving backups that has full resolution of the time series database.
11. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
monitoring, by a data protection system that includes one or more processors, a consolidation policy of the time series database, wherein consolidation policy includes consolidation data that identifies when data in the time series database is downscaled, the data including time series data, wherein the data in the time series database loses resolution when the data is downscaled;
defining, by the one or more processors, a backup policy of the data protection system based on the consolidation policy of the time series database;
generating, by the one or more processors, backups of the data in the time series database in accordance with the backup policy by the data protection system, wherein the backup policy ensures that the backups are generated before the data in the time series database is downscaled or discarded such that the backups contain full resolution copies of the time series data, wherein the backups further include backups of downscaled versions of the data in the time series database, wherein the downscaled versions have different resolutions of the data in the time series database; and
updating, by the one or more processors, a catalog that identifies where the backups are stored and resolutions of the backups.
12. The non-transitory storage medium of claim 11 , further comprising tracking changes to the consolidation policy, wherein the consolidation policy is read continuously, periodically, or when instructed.
13. The non-transitory storage medium of claim 11 , further comprising detecting, by the one or more processors, a change in the consolidation policy and dynamically adjusting the backup policy whenever the consolidation policy changes in order to accommodate the detected change to the consolidation policy.
14. The non-transitory storage medium of claim 11 , wherein different data sets in the time series data have different consolidation policies.
15. The non-transitory storage medium of claim 11 , further comprising monitoring usage of the backups and adjusting a retention period of the backups based on the monitored usage, wherein the retention period is based on user preferences and wherein the user preferences express a boundary for the retention period.
16. The non-transitory storage medium of claim 11 , further comprising adjusting a backup frequency when a time that the time series data is kept at a high resolution changes.
17. The non-transitory storage medium of claim 11 , further comprising creating a catalog to include in the backups, wherein the catalog identifies locations of the data and a resolution of the data.
18. The non-transitory storage medium of claim 11 , further comprising ensuring that a full resolution of the data is backed up before older data is downscaled.
19. The non-transitory storage medium of claim 11 , further comprising retraining a machine learning model based on older data that is at full resolution or older data that is at less than full resolution.
20. The non-transitory storage medium of claim 11 , wherein the backups include full resolutions of the data stored in the time series database, further comprising archiving backups that has full resolution of the time series database.