Historical network data-based scanning thread generation
Embodiments disclosed herein address the need to more efficiently backup a network-based storage environment that may be remote from a primary storage environment. For example, embodiments herein can provide a more efficient backup of a storage managed by a third-party entity. To improve the backup process, embodiments herein may optimize the number of scanning threads that are used to identify files that are to be backed up by, for example, analyzing the characteristics of the network and/or the network storage system to determine a number of scanning threads that will enable faster scanning of the network storage system while at the same time not overburden or be slowed down by a network between the network-based storage environment and the primary storage environment.
1. A method for performing multi-threaded scanning of a network storage system, the method comprising:
determining current network characteristics of a network used by one or more client computing devices to communicate with a network storage system;
determining an amount of scanning threads to initiate, based on an aggregation of historical network characteristics and the current network characteristics, wherein:
the historical network characteristics were previously obtained by performing one or more prior scans of the network storage system,
the historical network characteristics are weighted according to a recency in time of when the historical network characteristics were obtained,
more recent historical network characteristics are weighted heavier than less recent historical network characteristics,
recency in time for weighting the historical network characteristics is measured from a time when a corresponding backup process was performed on the network storage system for corresponding historical network characteristics,
the historical network characteristics and the current network characteristics are weighted differently in the aggregation of the historical network characteristics and the current network characteristics, and
the scanning threads are configured to scan a network storage repository of the network storage system to identify files to back up that are stored at the network storage repository;
causing the amount of the scanning threads to be initiated at the network storage system; and
receiving a scan result from the network storage system corresponding to a result of scanning the network storage repository using the amount of the scanning threads.
2. The method of claim 1 , wherein scanning the network storage repository comprises traversing a file structure at the network storage repository.
3. The method of claim 1 , further comprising:
initiating backup of the network storage repository based at least in part on the scan result.
4. The method of claim 3 , wherein the scan result identifies files to be backed up and at least a location of the files at the network storage repository.
5. The method of claim 3 , further comprising:
receiving files to be backed up in response to the initiating of the backup of the network storage repository; and
providing the files to a media agent for backup to a secondary storage system.
6. The method of claim 1 , wherein the historical network characteristics are determined based at least in part on communication with the network storage system during performance of a particular number of backup processes occurring during a particular time period prior to the determining of the current network characteristics.
7. The method of claim 1 , wherein the historical network characteristics are determined based at least in part on transmission of a number of test packets during a time period occurring prior to the determining of the current network characteristics.
8. A system for performing multi-threaded scanning of a network storage system, the system comprising:
one or more non-transitory, computer-readable media having computer-executable instructions stored thereon; and
one or more processors that, having executed the computer-executable instructions, configure the system to perform a plurality of operations comprising:
determining current network characteristics of a network used by one or more client computing devices to communicate with a network storage system;
determining an amount of scanning threads to initiate, based on an aggregation of historical network characteristics and the current network characteristics, wherein:
the historical network characteristics were previously obtained by performing one or more prior scans of the network storage system,
the historical network characteristics are weighted according to a recency in time of when the historical network characteristics were obtained,
more recent historical network characteristics are weighted heavier than less recent historical network characteristics,
recency in time for weighting the historical network characteristics is measured from a time when a corresponding backup process was performed on the network storage system for corresponding historical network characteristics,
the historical network characteristics and the current network characteristics are weighted differently in the aggregation of the historical network characteristics and the current network characteristics, and
the scanning threads are configured to scan a network storage repository of the network storage system to identify files to back up that are stored at the network storage repository;
causing the amount of the scanning threads to be initiated at the network storage system; and
receiving a scan result from the network storage system corresponding to a result of scanning the network storage repository using the amount of the scanning threads.
9. The system of claim 8 , wherein scanning the network storage repository comprises traversing a file structure at the network storage repository.
10. The system of claim 8 , wherein the plurality of operations further comprises:
initiating backup of the network storage repository based at least in part on the scan result.
11. The system of claim 10 , wherein the scan result identifies files to be backed up and at least a location of the files at the network storage repository.
12. The system of claim 10 , wherein the plurality of operations further comprises:
receiving files to be backed up in response to the initiating of the backup of the network storage repository; and
providing the files to a media agent for backup to a secondary storage system.
13. The system of claim 8 , wherein the historical network characteristics are determined based at least in part on communication with the network storage system during performance of a particular number of backup processes occurring during a particular time period prior to the determining of the current network characteristics.
14. The system of claim 8 , wherein the historical network characteristics are determined based at least in part on transmission of a number of test packets during a time period occurring prior to the determining of the current network characteristics.
15. A non-transitory, computer-readable medium having computer-executable instructions stored thereon that, when executed by one or more hardware processors, causes a system to perform a plurality of operations comprising:
determining current network characteristics of a network used by one or more client computing devices to communicate with a network storage system;
determining an amount of scanning threads to initiate, based on an aggregation of historical network characteristics and the current network characteristics, wherein:
the historical network characteristics were previously obtained by performing one or more prior scans of the network storage system,
the historical network characteristics are weighted according to a recency in time of when the historical network characteristics were obtained,
more recent historical network characteristics are weighted heavier than less recent historical network characteristics,
recency in time for weighting the historical network characteristics is measured from a time when a corresponding backup process was performed on the network storage system for corresponding historical network characteristics,
the historical network characteristics and the current network characteristics are weighted differently in the aggregation of the historical network characteristics and the current network characteristics, and
the scanning threads are configured to scan a network storage repository of the network storage system to identify files to back up that are stored at the network storage repository;
causing the amount of the scanning threads to be initiated at the network storage system; and
receiving a scan result from the network storage system corresponding to a result of scanning the network storage repository using the amount of the scanning threads.
16. The non-transitory, computer-readable medium of claim 15 , wherein scanning the network storage repository comprises traversing a file structure at the network storage repository.
17. The non-transitory, computer-readable medium of claim 15 , wherein the plurality of operations further comprises:
initiating backup of the network storage repository based at least in part on the scan result.
18. The non-transitory, computer-readable medium of claim 17 , wherein the scan result identifies files to be backed up and at least a location of the files at the network storage repository.
19. The non-transitory, computer-readable medium of claim 17 , wherein the plurality of operations further comprises:
receiving files to be backed up in response to the initiating of the backup of the network storage repository; and
providing the files to a media agent for backup to a secondary storage system.
20. The non-transitory, computer-readable medium of claim 15 , wherein the historical network characteristics are determined based at least in part on communication with the network storage system during performance of a particular number of backup processes occurring during a particular time period prior to the determining of the current network characteristics.