Optimized filesystem walk for backup operations
The system for backing up data comprises an input interface configured to receive an input set of files. The system for backing up data additionally comprises a processor configured to add modified objects to a list of modified objects in the set of files, and, in the event it is determined to provide the list of modified objects to a backup process, provide the list to the backup process.
1. A method comprising:
receiving a set of files via an input interface;
adding, by one or more processors, a modified object to one or more lists of modified objects, the modified object being among a plurality of objects that are to be backed up, wherein a modified object is a data structure that aggregates electronic data;
adding the modified object to the one or more lists of modified objects comprising:
determining that the modified object is not within a defined proximity of other modified objects included in at least one corresponding list of modified objects, wherein the defined proximity corresponds to each modified object's location in a storage device or memory; and
in response to determining that the modified object is not within the defined proximity of the other modified objects included in the corresponding at least one list of modified objects, creating a new list of modified objects and adding the modified object to the new list of modified objects; and
performing a backup process for the one or more lists of modified objects, wherein the backup process processes the new list in connection with backing up one or more of the modified objects.
2. The method of claim 1 , wherein the performing the backup process with respect to the one or more lists of modified objects comprises:
performing a backup with respect to objects included in at least one list of modified objects among the one or more lists of modified objects.
3. The method of claim 2 , wherein the at least one list of modified objects is determined based at least in part on one or more of a threshold amount of data or a threshold number of files included in the list of modified objects.
4. The method of claim 3 , further comprising:
determining whether the at least one list of modified objects satisfies one or more conditions pertaining to the threshold amount of data and the threshold number of files included in the list of modified objects; and
in response to determining that the at least one list of modified objects satisfies the one or more conditions pertaining to the threshold amount of data and the threshold number of files included in the list of modified objects, passing the at least one list of modified objects to the backup process.
5. The method of claim 1 , wherein the performing the backup process with respect to the one or more lists of modified objects is based at least in part on a total file size of a corresponding one of the one or more lists of modified objects.
6. The method of claim 1 , wherein the backup process queues the modified objects in the list to be read.
7. The method of claim 6 , wherein the performing the backup process with respect to the one or more lists of modified objects comprises reading modified objects in at least one list of the one or more lists in a single access to storage device.
8. The method of claim 6 , wherein the performing the backup process with respect to the one or more lists of modified objects comprises reading modified objects in at least one list of the one or more lists in multiple reads in a sequence.
9. The method of claim 8 , wherein the sequence targets a minimum seek time.
10. The method of claim 8 , wherein the sequence targets a minimum access time.
11. The method of claim 1 , wherein the modified object is identified from among a set of files comprising one or more of the following: a complete system, a list of filesystems, a single filesystem, a mount point, a logical object that is used to aggregate files, or a folder.
12. The method of claim 1 , comprising determine whether to provide the new list of modified objects to the backup process based at least in part on one or more of the threshold amount of data or the threshold number of files included in the new list of modified objects.
13. The method of claim 1 , wherein the one or more lists of modified objects and the new list of modified objects are associated with different backup buffers.
14. The method of claim 1 , wherein determining, in connection with walking of a set of files, whether to add the modified object to the one or more lists of modified objects based on whether the modified object has changed since a last backup.
15. The method of claim 1 , wherein determining that the modified object is not within a defined proximity of other modified objects included in a corresponding at least one list of modified objects includes determining that the modified object is from a same volume as the other modified objects included on the at least one list of modified objects.
16. The method of claim 1 , wherein determining that the modified object is not within a defined proximity of other modified objects included in a corresponding at least one list of modified objects includes determining whether the modified object is stored on a volume that is the same as one or more volumes on which the other modified objects included in the at least one list of modified objects are stored.
17. The method of claim 1 , further comprising determining whether to add the modified object to the one or more lists of modified objects based at least in part on a determination of whether a backup storage system comprises the corresponding modified object.
18. The method of claim 1 , wherein the backup process queues the modified objects in the corresponding at least one list of modified objects such that the backup process reads the modified objects from storage in a sequential manner based on a corresponding physical storage locations of the plurality of objects.
19. A system, comprising:
an input interface configured to receive a set of files; and
a processor configured to:
add a modified object to one or more lists of modified objects, the modified object being among a plurality of objects that are to be backed up, wherein a modified object is a data structure that aggregates electronic data;
determine that the modified object is not within a defined proximity of other modified objects, wherein the defined proximity corresponds to each modified object's location in a storage device or memory; and
in response to determining that the modified object is not within the defined proximity of the other modified objects, create a new list of modified objects and add the modified object to the new list of modified objects; and
perform a backup process for the one or more lists of modified objects, wherein the backup process processes the new list in connection with backing up one or more of the modified objects.
20. A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
receiving a set of files via an input interface;
adding, by one or more processors, a modified object to one or more lists of modified objects, the modified object being among a plurality of objects that are to be backed up, wherein a modified object is a data structure that aggregates electronic data;
adding the modified object to the one or more lists of modified objects comprising:
determining that the modified object is not within a defined proximity of other modified objects; and
in response to determining that the modified object is not within the defined proximity of the other modified objects, creating a new list of modified objects and adding the modified object to the new list of modified objects, wherein the defined proximity corresponds to each modified object's location in a storage device or memory; and
performing a backup process for the one or more lists of modified objects, wherein the backup process processes the new list in connection with backing up one or more of the modified objects.