Network storage backup using distributed media agents
Certain embodiments disclosed herein reduce or eliminate a communication bottleneck at the storage manager by reducing communication with the storage manager while maintaining functionality of an information management system. In some implementations, a client obtains information for enabling a secondary storage job (e.g., a backup or restore) from a storage manager and stores the information (which may be referred to as job metadata) in a local cache. The client may then reuse the job metadata for multiple storage jobs reducing the frequency of communication with the storage manager. When a configuration of the information management system changes, or the availability of resources changes, the storage manager can push updates to the job metadata to the clients. Further, a client can periodically request updated job metadata from the storage manager ensuring that the client does not rely on out-of-date job metadata.
1. A system comprising:
a first computing device that comprises one or more hardware processors, wherein the first computing device executes a data agent and a data migration system that configure the first computing device to:
detect that first data is to be backed up from a primary storage device to a network storage system that is distinct from the primary storage device,
wherein the first data is generated at the first computing device and is stored in a first data format at the primary storage device;
based on making a determination that the first computing device is capable of interacting with the network storage system, perform a backup process that configures the first computing device to:
obtain the first data from the primary storage device,
transmit the first data, in a backup format, from the first computing device to the network storage system, and
cause the network storage system to store the first data in the backup format,
wherein the backup format is distinct from the first data format; and
wherein the backup process further configures the first computing device to transmit metadata generated by the backup process to a second computing device, which is distinct from the first computing device, and wherein the second computing device is configured to generate a backup index, wherein at least some of the backup index is based on the metadata generated by the backup process at the first computing device.
2. The system of claim 1 , wherein the backup index indicates where the first data is stored at the network storage system.
3. The system of claim 1 , wherein the second computing device is further configured to: generate one or more transaction log files that are based on the metadata generated by the backup process at the first computing device, wherein each transaction log file comprises one or more attributes of at least portions of the first data that was backed up by the backup process.
4. The system of claim 3 , wherein the second computing device is further configured to: back up the one or more transaction log files to the network storage system.
5. The system of claim 1 , wherein the first computing device is further configured to: use a job identifier to perform the backup process, and use the job identifier to perform at least one subsequent backup process without obtaining another job identifier for the at least one subsequent backup process.
6. The system of claim 1 , wherein the first computing device is further configured to: obtain a job identifier from a storage manager computing device, store the job identifier at the first computing device, use the job identifier to perform the backup process, and use the job identifier to perform at least one subsequent backup process without obtaining another job identifier from the storage manager computing device for the at least one subsequent backup process.
7. The system of claim 1 , wherein the network storage system is configured in a cloud-based storage system.
8. The system of claim 1 , wherein the first computing device is further configured to: based on making a determination that the first computing device is not capable of interacting with the network storage system, perform a second backup process at the first computing device, wherein the second backup process configures the first computing device to: obtain the first data from the primary storage device, and transmit the first data from the first computing device to the second computing device; and
wherein the second computing device is configured to: transmit the first data, in a backup format, from the second computing device to the network storage system, and cause the network storage system to store the first data in the backup format.
9. A computer-implemented method, performed by one or more computing devices that comprise one or more hardware processors, which execute computer programming instructions, the method comprising:
by a first computing device among the one or more computing devices:
detecting a trigger to back up first data from a primary storage device to a network storage system that is distinct from the primary storage device,
wherein the first data is generated at the first computing device and is stored in a primary data format at the primary storage device;
based on making a determination that the first computing device is capable of interacting with the network storage system, performing a first backup process comprising:
obtaining the first data from the primary storage device,
transmitting the first data, in a backup format, from the first computing device to the network storage system,
causing the network storage system to store the first data in the backup format,
wherein the backup format is distinct from the primary data format,
causing the first computing device not to transmit the first data to a second computing device that is distinct from the first computing device, and
transmitting, to the second computing device, metadata generated by the first backup process; and
by the second computing device among the one or more computing devices: generating a backup index, wherein at least some of the backup index is based on the metadata generated at the first computing device.
10. The computer-implemented method of claim 9 further comprising:
based on making a determination that the first computing device is not capable of interacting with the network storage system, performing a second backup process comprising: obtaining the first data from the primary storage device, and transmitting, to the second computing device, the first data and metadata generated by the second backup process; and
by the second computing device among the one or more computing devices:
transmitting the first data, in a backup format, from the second computing device to the network storage system, and
causing the network storage system to store the first data in the backup format.
11. The computer-implemented method of claim 9 , wherein the backup index indicates where the first data is stored at the network storage system.
12. The computer-implemented method of claim 9 further comprising: by the second computing device among the one or more computing devices: generating one or more transaction log files that are based on the metadata generated at the first computing device, wherein each transaction log file comprises one or more attributes of at least portions of the first data that was backed up by the first backup process.
13. The computer-implemented method of claim 12 further comprising: by the second computing device among the one or more computing devices: backing up the one or more transaction log files to the network storage system.
14. The computer-implemented method of claim 9 further comprising: by the first computing device: using a job identifier to perform the first backup process, and using the job identifier to perform at least one subsequent first backup process without obtaining another job identifier for the at least one subsequent first backup process.
15. The computer-implemented method of claim 9 further comprising: by the first computing device: obtaining a job identifier from a storage manager computing device, among the one or more computing devices, storing the job identifier at the first computing device, using the job identifier to perform the first backup process, and using the job identifier to perform at least one subsequent first backup process without obtaining another job identifier from the storage manager computing device for the at least one subsequent first backup process.
16. The computer-implemented method of claim 9 , wherein the network storage system is configured in a cloud-based storage system.