Dynamic triggering of block-level backups based on block change thresholds and corresponding file identities using indexing in a data storage management system
A data storage management approach is disclosed that performs backup operations flexibly, based on a dynamic scheme of monitoring block changes occurring in production data. The illustrative system monitors block changes based on certain block-change thresholds and triggers block-level backups of the changed blocks when a threshold is passed. Block changes may be monitored in reference to particular files based on a reverse lookup mechanism. The illustrative system also collects and stores historical information on block changes, which may be used for reporting and predictive analysis.
1. A method for triggering block-level backups based on block change thresholds and corresponding file identities, the method comprising:
receiving, by a first computing device, a threshold measure of changed data blocks, wherein the first computing device is configured to monitor changed data blocks against the threshold measure;
receiving, by the first computing device, a plurality of point-in-time representations of changed data blocks stored in one or more first data storage devices,
wherein the changed data blocks result from write operations performed by an application that executes on a second computing device in communication with the one or more first data storage devices,
wherein the plurality of point-in-time representations of changed data blocks are received from a first data agent that tracks data block changes generated by the application, and
wherein the first data agent executes on the second computing device;
by the first computing device, receiving from a second data agent, file-to-data-block relationship information that identifies second data blocks that store contents of a first data file accessible to the application,
wherein the second data agent tracks a file system comprising one or more data files accessible to the application including the first data file, and
wherein the second data agent executes on the second computing device;
by the first computing device, indexing the file-to-data-block relationship information into a data-block-to-file mapping that identifies the first data file based on a given one of the second data blocks;
determining, by the first computing device, that the threshold measure has been passed relative to the first data file by a first set of changed data blocks on the one or more first data storage devices,
by (i) analyzing the plurality of point-in-time representations of changed data blocks received from the first data agent and (ii) further by determining based on the data-block-to-file mapping that the first set of changed data blocks correspond to the first data file; and
based on determining that the threshold measure has been passed relative to the first data file,
causing, by the first computing device, a block-level backup of the first set of changed data blocks to be performed at least in part by the first data agent, wherein a secondary copy that results from the block-level backup is associated with the first data file and not with other data files accessible to the application.
2. The method of claim 1 , wherein the block-level backup of the first set of changed data blocks generates at least one secondary copy of the first set of changed data blocks but does not back up the first data file in its entirety.
3. The method of claim 1 ,
wherein the second data agent obtains the file-to-data-block relationship information by executing a scan of the file system it tracks.
4. The method of claim 1 further comprising:
receiving, by the first computing device, application-to-file relationship information about the one or more data files that are accessible to the application;
indexing, by the first computing device, the received application-to-file relationship information into a file-to-application mapping; and
identifying, by the first computing device, based on the application-to-file mapping and the data-block-to-file mapping, the application that changed the first set of changed data blocks.
5. The method of claim 1 wherein the threshold measure identifies the first data file accessible to the application.
6. The method of claim 1 wherein the first data agent is a component of a data storage management system that is managed by a storage manager; and
wherein the causing of the block-level backup comprises:
transmitting, by the first computing device to the storage manager, an indication that the threshold measure of changed data blocks has been passed by the first set of changed data blocks,
initiating the block-level backup, by the storage manager, which comprises instructing the first data agent and a media agent to perform the block-level backup of the first set of changed data blocks,
storing the secondary copy that results from the block-level backup to a second data storage device,
associating the secondary copy with the first data file, and
wherein the media agent and the second data storage device are also components of the data storage management system.
7. The method of claim 6 wherein the threshold measure is received by the first computing device from the storage manager.
8. The method of claim 1 wherein the threshold measure measures changed data blocks over a pre-determined period of time; and
wherein the block-level backup of the first set of data blocks is based on changes during the pre-determined period of time.
9. The method of claim 1 wherein the threshold measure measures changed data blocks in a storage volume in the one or more first data storage devices;
wherein the first set of changed data blocks are stored in the storage volume and not elsewhere in the one or more first data storage devices; and
wherein the block-level backup of the first set of changed data blocks is based on changes to the storage volume.
10. The method of claim 1 wherein the threshold measure measures a rate of change of data blocks.
11. The method of claim 1 wherein the threshold measure measures a rate of change of data blocks to the first data file among the one or more data files accessible to the application;
wherein the first set of changed data blocks store contents of the first data file but do not store contents of other of the one or more data files accessible to the application; and
wherein the block-level backup of the first set of changed data blocks is based on the rate of change to the first data file.
12. A non-transitory computer-readable medium storing instructions that, when executed by a first computing device comprising one or more hardware processors, cause the first computing device to perform operations for triggering block-level backups based on block change thresholds and corresponding file identities, the operations comprising:
receiving a threshold measure of changed data blocks;
monitoring changed data blocks against the threshold measure;
receiving from a first data agent a plurality of point-in-time representations of changed data blocks stored in one or more first data storage devices,
wherein the changed data blocks result from write operations performed by an application that executes on a second computing device in communication with the one or more first data storage devices,
wherein the first data agent tracks data block changes generated by the application, and
wherein the first data agent executes on the second computing device;
receiving from a second data agent file-to-data-block relationship information that identifies second data blocks that store contents of a first data file accessible to the application,
wherein the second data agent tracks a file system comprising one or more data files accessible to the application including the first data file, and
wherein the second data agent executes on the second computing device;
indexing the file-to-data-block relationship information into a data-block-to-file mapping that identifies the first data file based on a given one of the second data blocks;
based on (i) the plurality of point-in-time representations of changed data blocks received from the first data agent, and (ii) the data-block-to-file mapping indicating that a first set of changed data blocks correspond to the first data file, determining that the threshold measure has been passed relative to the first data file by the first set of changed data blocks on the one or more first data storage devices; and
based on determining that the threshold measure has been passed relative to the first data file, causing a block-level backup of the first set of changed data blocks to be performed at least in part by the first data agent,
wherein a secondary copy that results from the block-level backup is associated with the first data file and not with other files accessible to the application.
13. The non-transitory computer-readable medium of claim 12 , wherein the threshold measure measures changed data blocks over a pre-determined period of time; and
wherein the block-level backup of the first set of changed data blocks is based on changes during the pre-determined period of time.
14. The non-transitory computer-readable medium of claim 12 , wherein the threshold measure measures changed data blocks in a storage volume in the one or more first data storage devices;
wherein the first set of changed data blocks are stored in the storage volume and not elsewhere in the one or more first data storage devices; and
wherein the block-level backup of the first set of changed data blocks is based on changes to the storage volume.
15. The non-transitory computer-readable medium of claim 12 , wherein the threshold measure measures a rate of change of data blocks.
16. The non-transitory computer-readable medium of claim 12 , wherein the threshold measure measures a rate of change of data blocks to the first data file among the one or more data files accessible to the application;
wherein the first set of changed data blocks store contents of the first data file but do not store contents of other of the one or more data files accessible to the application; and
wherein the block-level backup of the first set of changed data blocks is based on the rate of change to the first data file.
17. The non-transitory computer-readable medium of claim 12 , wherein the block-level backup of the first set of changed data blocks generates at least one secondary copy of the first set of changed data blocks but does not back up the first data file in its entirety.
18. The non-transitory computer-readable medium of claim 12 , wherein causing the block-level backup comprises:
transmitting to a storage manager, an indication that the threshold measure of changed data blocks has been passed by the first set of changed data blocks;
wherein the block-level backup is initiated by the storage manager, comprising instructing the first data agent and a media agent to perform the block-level backup of the first set of changed data blocks; and
wherein the storage manager executes on a computing device that is distinct from the first computing device.
19. The non-transitory computer-readable medium of claim 18 , wherein the threshold measure is received by the first computing device from the storage manager.
20. The non-transitory computer-readable medium of claim 12 , wherein causing the block-level backup comprises causing the secondary copy to be associated with the first data file.