Multi-threaded scanning of distributed file systems
Embodiments disclosed herein address the need to more efficiently backup a network-based storage environment that may be remote from a primary storage environment. For example, embodiments herein can provide a more efficient backup of a storage managed by a third-party entity. To improve the backup process, embodiments herein may optimize the number of scanning threads that are used to identify files that are to be backed up by, for example, analyzing the characteristics of the network and/or the network storage system to determine a number of scanning threads that will enable faster scanning of the network storage system while at the same time not overburden or be slowed down by a network between the network-based storage environment and the primary storage environment.
1. A non-transitory, computer-readable medium storing computer-executable instructions which, when executed by one or more processors, cause a computing system to perform a method for backing up one or more data files, wherein the method comprises:
determining characteristics of a network that is used to communicate with a network storage repository;
determining an amount of scanning threads to use for scanning the network storage repository, wherein the amount of scanning threads is based on an amount of time that is expected to complete the scanning of the network storage repository using the amount of scanning threads and is further based on an amount of resources available within the network to communicate results of the scanning;
initiating the amount of scanning threads for scanning the network storage repository, wherein the amount of scanning threads are configured to identify one or more data files that are to be backed up from the network storage repository; and
based on the results of the scanning of the network storage repository, initiating a backup of the one or more data files.
2. The non-transitory, computer-readable medium of claim 1 , wherein the network storage repository is cloud-based.
3. The non-transitory, computer-readable medium of claim 1 , wherein a result of the scanning identifies corresponding locations, at the network storage repository, of the one or more data files to be backed up.
4. The non-transitory, computer-readable medium of claim 1 , wherein the method further comprises:
providing the one or more data the files to a media agent for storing backup copies of the one or more data files at a secondary storage system that is distinct from the network storage repository.
5. The non-transitory, computer-readable medium of claim 1 , wherein:
a data agent executing on a first computing device performs the determining of the amount of scanning threads for scanning the network storage repository, and
the data agent causes the initiating of the amount of scanning threads to identify the one or more data files.
6. The non-transitory, computer-readable medium of claim 5 , wherein:
a storage manager initiates the backup of the one or more data files, and
the storage manager executes on a second computing device that is distinct from the first computing device.
7. The non-transitory, computer-readable medium of claim 1 , wherein the scanning of the network storage repository comprises traversing a file structure at the network storage repository.
8. A computer-implemented method for backing up one or more data files, the method comprising:
determining characteristics of a network that is used to communicate with a network storage repository;
determining an amount of scanning threads to use for scanning a network storage repository, wherein the amount of scanning threads is based on an amount of time that is expected to complete the scanning of the network storage repository using the amount of scanning threads and is further based on an amount of resources available within the network to communicate results of the scanning;
initiating the amount of scanning threads to perform the scanning of the network storage repository, wherein the scanning identifies one or more data files to be backed up from the network storage repository; and
based on the results of the scanning, initiating a backup of the one or more data files.
9. The method of claim 8 , wherein the network storage repository is cloud-based.
10. The method of claim 8 , wherein a result of the scanning identifies corresponding locations, at the network storage repository, of the one or more data files to be backed up.
11. The method of claim 8 , further comprising:
providing the one or more data the files to a media agent for storing backup copies of the one or more data files at a secondary storage system that is distinct from the network storage repository.
12. The method of claim 8 , wherein:
a data agent executing on a first computing device performs the determining of the amount of scanning threads for scanning the network storage repository, and
the data agent causes the initiating of the amount of scanning threads.
13. The method of claim 12 , wherein:
a storage manager initiates the backup of the one or more data files, and
the storage manager executes on a second computing device that is distinct from the first computing device.
14. The method of claim 8 , wherein the scanning of the network storage repository comprises traversing a file structure at the network storage repository.
15. A system for backing up one or more data files comprising:
one or more non-transitory, computer-readable media having computer-executable instructions stored thereon; and
one or more processors that, having executed the computer-executable instructions, configure the system to perform a plurality of operations comprising:
determining characteristics of a network that is used to communicate with a network storage repository;
determining an amount of scanning threads to be used for scanning the network storage repository, wherein the amount of scanning threads is based on an amount of time that is expected to complete the scanning of the network storage repository using the amount of scanning threads and is further based on an amount of resources available within the network to communicate results of the scanning;
initiating the amount of scanning threads to perform the scanning of the network storage repository, wherein the scanning identifies one or more data files to be backed up from the network storage repository; and
based on the results of the scanning, initiating a backup of the one or more data files.
16. The system of claim 15 , wherein the network storage repository is cloud-based.
17. The system of claim 15 , wherein a result of the scanning identifies corresponding locations, at the network storage repository, of the one or more data files to be backed up.
18. The system of claim 15 , wherein the plurality of operations further comprises:
providing the one or more data the files to a media agent for storing backup copies of the one or more data files at a secondary storage system.
19. The system of claim 15 , wherein:
a data agent executing on a first computing device performs the determining of the amount of scanning threads for scanning the network storage repository, and
the data agent causes the initiating of the amount of scanning threads.
20. The system of claim 19 , wherein:
a storage manager initiates the backup of the one or more data files, and
the storage manager executes on a second computing device that is distinct from the first computing device.