Failover systems and methods for performing backup operations
In certain embodiments, a tiered storage system is disclosed that provides for failover protection during data backup operations. The system can provide for an index, or catalog, for identifying and enabling restoration of backup data located on a storage device. The system further maintains a set of transaction logs generated by media agent modules that identify metadata with respect to individual data chunks of a backup file on the storage device. A copy of the catalog and transaction logs can be stored at a location accessible by each of the media agent modules. In this manner, in case of a failure of one media agent module during backup, the transaction logs and existing catalog can be used by a second media agent module to resume the backup operation without requiring a restart of the backup process.
1. A method for performing an operation in a data storage system, the method comprising:
with a first computing device comprising one or more hardware processors:
receiving a plurality of data units from a client computing device to store on at least one first storage device as part of a data protection operation;
storing at least a first data unit of the plurality of data units on the at least one first storage device; and
prior to completion of the data protection operation, storing metadata to at least one second storage device in association with the storing of the first data unit and prior to, concurrently with, or subsequent to the storing of the first data unit, wherein the at least one second storage device is accessible by the first computing device and at least one other computing device; and
with a second computing device comprising one or more hardware processors:
receiving an instruction to take over control of the data protection operation partially performed, but not yet completed, by the first computing device;
obtaining, from the at least one second storage device, the metadata associated with the storing of the first data unit on the at least one first storage device;
using at least the metadata, determining a point in the data protection operation at which the first computing device ceased performance of the data protection operation;
using the determined point in the data protection operation to resume performance of the data protection operation at least by storing on the at least one first storage device at least a second data unit of the plurality of data units that has not yet been stored on the at least one first storage device, and without repeating the storage of the first data unit on the at least one first storage device; and
updating the metadata stored to the at least one second storage device after completing the data protection operation,
wherein the at least one second storage device is maintained by a third computing device comprising one or more hardware processors and in networked communication with the first and second computing devices, the third computing device configured to instruct the second computing device to take over control of the data protection operation.
2. The method of claim 1 , further comprising, with the second computing device and prior to resuming performance of the data protection operation:
obtaining an index associated with the data protection operation;
applying the metadata to the index to create an updated index; and
storing the updated index.
3. The method of claim 1 , wherein the storing of the metadata comprises generating a transaction log comprising the metadata and storing the transaction log to the at least one second storage device.
4. The method of claim 1 , wherein the third computing device is configured to instruct the second computing device to take over control of the data protection operation in response to detecting that the first computing device has prematurely ceased performing the data protection operation.
5. The method of claim 1 , wherein the third computing device is configured to instruct the second computing device to take over control of the data protection operation in response to detecting that the first computing device has a load that exceeds a threshold level.
6. The method of claim 1 , wherein the third computing device is configured to instruct the second computing device to take over control of the data protection operation according to a schedule associated with the first computing device.
7. A data storage system for performing an operation, comprising:
a first computing device comprising one or more hardware processors, wherein the first computing device is configured to:
receive a plurality of data units from a client computing device to store on at least one first storage device as part of a data protection operation;
store at least a first data unit of the plurality of data units on the at least one first storage device; and
prior to completion of the data protection operation, storing metadata to at least one second storage device in association with the storing of the first data unit and prior to, concurrently with, or subsequent to the storing of the first data unit, wherein the at least one second storage device is accessible by the first computing device and at least one other computing device;
a second computing device comprising one or more hardware processors, wherein the second computing device is configured to:
receive an instruction to take over control of the data protection operation partially performed, but not yet completed, by the first computing device;
obtain, from the at least one second storage device, the metadata associated with the storing of the first data unit on the at least one first storage device;
using at least the metadata, determine a point in the data protection operation at which the first computing device ceased performance of the data protection operation;
using the determined point in the data protection operation to resume performance of the data protection operation at least by storing on the at least one first storage device at least a second data unit of the plurality of data units that has not yet been stored on the at least one first storage device, and without repeating the storage of the first data unit on the at least one first storage device; and
update the metadata stored to the at least one second storage device after completing the data protection operation; and
a third computing device comprising one or more hardware processors and in networked communication with the first and second computing devices, wherein the third computing device is configured to maintain the at least one second storage device and to instruct the second computing device to take over control of the data protection operation.
8. The data storage system of claim 7 , wherein the second computing device is further configured to, prior to resuming performance of the data protection operation:
obtain an index associated with the data protection operation;
apply the metadata to the index to create an updated index; and
store the updated index.
9. The data storage system of claim 7 , wherein the first computing device is configured to store the metadata by generating a transaction log comprising the metadata and storing the transaction log to the at least one second storage device.
10. The data storage system of claim 7 , wherein the third computing device is configured to instruct the second computing device to take over control of the data protection operation in response to detecting that the first computing device has prematurely ceased performing the data protection operation.
11. The data storage system of claim 7 , wherein the third computing device is configured to instruct the second computing device to take over control of the data protection operation in response to detecting that the first computing device has a load that exceeds a threshold level.
12. The data storage system of claim 7 , wherein the third computing device is configured to instruct the second computing device to take over control of the data protection operation according to a schedule associated with the first computing device.
13. A non-transitory computer readable medium comprising code that, when executed, causes:
a first computing device comprising one or more hardware processors to:
receive a plurality of data units from a client computing device to store on at least one first storage device as part of a data protection operation;
store at least a first data unit of the plurality of data units on the at least one first storage device; and
prior to completion of the data protection operation, storing metadata to at least one second storage device in association with the storing of the first data unit and prior to, concurrently with, or subsequent to the storing of the first data unit, wherein the at least one second storage device is accessible by the first computing device and at least one other computing device;
a second computing device comprising one or more hardware processors to:
receive an instruction to take over control of the data protection operation partially performed, but not yet completed, by the first computing device;
obtain, from the at least one second storage device, the metadata associated with the storing of the first data unit on the at least one first storage device;
using at least the metadata, determine a point in the data protection operation at which the first computing device ceased performance of the data protection operation;
using the determined point in the data protection operation to resume performance of the data protection operation at least by storing on the at least one first storage device at least a second data unit of the plurality of data units that has not yet been stored on the at least one first storage device, and without repeating the storage of the first data unit on the at least one first storage device; and
update the metadata stored to the at least one second storage device after completing the data protection operation; and
a third computing device comprising one or more hardware processors and in networked communication with the first and second computing devices to maintain the at least one second storage device and to instruct the second computing device to take over control of the data protection operation.
14. The computer readable medium of claim 13 , wherein the code further causes the second computing device to, prior to resuming performance of the data protection operation:
obtain an index associated with the data protection operation;
apply the metadata to the index to create an updated index; and
store the updated index.
15. The computer readable medium of claim 13 , wherein the code causes the third computing device to instruct the second computing device to take over control of the data protection operation based on one of (i) detecting that the first computing device has prematurely ceased performing the data protection operation, (ii) detecting that the first computing device has a load that exceeds a threshold level, or (iii) a schedule associated with the first computing device.