Systems and methods for large scale complex storage operation execution
A Multi-Threaded Indexed (“MTI”) file system may use a first set of threads, processes, or executable instances to index desired file attributes in a database while simultaneously but independently executing file operations with a second set of threads, processes, or executable instances. In response to receiving a file operation, the second set of threads, processes, or executable instance may query the database to directly identify files that are indirectly implicated by the file operation with a wildcard, regular expression, and/or other expression that indirectly identifies the files based on different file attributes, paths, name expressions, or combinations thereof. The second set of threads, processes, or executable instances are therefore able to identify the files implicated by the file operation based solely on the indexed file attributes already entered in the database without the need to load and scan the metadata of files in directories targeted by the file operation.
1. A method comprising:
providing a file system that stores a plurality of files to at least one storage device and that creates an entry in a database for each file of the plurality of files, wherein each entry comprises an identifier that identifies a storage location of a different file on the at least one storage device;
tracking, by operation of the file system, at least one attribute from a set of the plurality of files based on the file system setting a particular index in each entry of a set of entries created in the database for the set of files;
receiving a file operation at the file system;
querying, by the file system, the database for a subset of the set of entries having the particular index set to one or more values that satisfy a condition of the file operation, wherein the condition is defined using the at least one attribute; and
performing, by the file system, the file operation against a subset of the set of files that is identified on the at least one storage device from the identifier associated with each entry of the subset of entries, and wherein performing the file operation comprises modifying the subset of files on the at least one storage device according to the file operation.
2. The method of claim 1 further comprising:
generating a first set of executable instances and an independently executing second set of executable instances within the file system;
setting the particular index in response to a first executable instance of the first set of executable instances running at a first interval or in response to a first set of events, wherein the particular index is a first index;
setting a second index in one or more of the set of entries in response to a second executable instance of the first set of executable instances running at a second interval, that is different than the first interval, or in response to a second set of events, that is different than the first set of events; and
wherein performing the file operation comprises:
initiating execution of the second set of executable instances in response to receiving the file operation;
mapping the condition from the file operation to the particular index; and
selecting the subset of files based on results of said querying and without inspecting metadata of the set of files stored on the at least one storage device.
3. The method of claim 1 ,
wherein said tracking comprises scanning metadata of the set of files, wherein the metadata comprises the at least one attribute; and
wherein said querying comprises selecting the subset of files based on the subset of entries identified from said querying independently of said tracking or said scanning.
4. The method of claim 1 ,
wherein said tracking is performed by execution of a first thread or process of the file system, and
wherein said querying and said performing is performed by execution of at least a second thread or process of the file system that runs simultaneously with the first thread or process.
5. The method of claim 1 further comprising:
scanning the at least one file attribute of new or modified files from the set of files periodically with a first set of executable threads of the file system; and
setting, by operation of the first set of executable threads, the particular index for an entry of a particular file in response to detecting the particular file in the set of files with the at least one attribute having one or more specified values.
6. The method of claim 5 further comprising:
generating at least one executable thread from a second set of executable threads of the file system that execute independently of the first set of executable threads in response to receiving the file operation; and
determining, by operation of the at least one executable thread of the second set of executable threads, the subset of files implicated by the file operation based on indexes in the database.
7. The method of claim 5 , wherein said setting the particular index for the particular file comprises:
updating, by operation of a first executable thread of the first set of executable threads, a value of the particular index for the entry of the particular file in response to the detecting that the particular file has a first attribute with a first value; and
updating, by operation of a second executable thread of the first set of executable threads, a value of a different second index for the entry of the particular file in response to the detecting that the particular file has a second attribute with a different second value, wherein the second executable thread starts execution at a different time than the first executable thread.
8. The method of claim 7 ,
wherein the first attribute is a time-to-live (“TTL”) attribute and the first value indicates that the TTL attribute for the particular file has expired; and
wherein the second attribute is a last access time and the second value indicates that the last access time is greater than a threshold access time.
9. The method of claim 1 further comprising:
obtaining, at the file system, a directory path and filename of each file of the subset of files from the identifier of each entry of the subset of entries identified from said querying.
10. The method of claim 9 , wherein modifying the subset of files comprises:
purging the subset of files in response to the file system issuing a delete command to the different path and filename of each file of the subset of files.
11. The method of claim 1 , wherein said tracking comprises:
identifying the set of files from a write queue of the file system, wherein the write queue comprises a list of modified files; and
scanning the at least one attribute of the set of files upon the set of files being written to the at least one storage device.
12. The method of claim 1 , wherein said tracking comprises:
setting a first index of the set of entries for a first file of the set of files in response to detecting that a first attribute of the first file has changed at a first time; and
setting a second index of the set of entries for a second file of the set of files in response to detecting that a second attribute of the second file has changed at a different second time.
13. The method of claim 1 further comprising:
updating a second index, that is different than the particular index, while simultaneously performing the file operation against the subset of files identified from said querying.
14. The method of claim 1 further comprising:
removing the set of entries from the database in response to deleting the subset of files from storage as a result of performing the file operation.
15. A device comprising:
at least one storage device;
a database;
a file system; and
one or more processors configured to:
store, by operation of the file system, a plurality of files to the at least one storage device while creating an entry in the database for each file of the plurality of files, wherein each entry comprises an identifier that identifies a storage location of a different file on the at least one storage device;
track at least one attribute from a set of the plurality of files based on the file system setting a particular index in each entry of a set of entries created in the database for the set of files;
receive a file operation at the file system;
query the database for a subset of the set of entries having the particular index set to one or more values that satisfy a condition of the file operation, wherein the condition is defined using the at least one attribute; and
perform the file operation against a subset of the set of files that is identified on the at least one storage device from the identifier associated with each entry of the subset of entries, and wherein performing the file operation comprises modifying the subset of files on the at least one storage device according to the file operation.
16. The device of claim 15 , wherein the one or more processors are further configured to:
generate a first set of executable instances and an independently executing second set of executable instances within the file system;
set the particular index in response to a first executable instance of the first set of executable instances running at a first interval or in response to a first set of events, wherein the particular index is a first index;
set a second index in one or more of the set of entries in response to a second executable instance of the first set of executable instances running at a second interval, that is different than the first interval, or in response to a second set of events, that is different than the first set of events; and
wherein performing the file operation comprises:
initiating execution of the second set of executable instances in response to receiving the file operation;
mapping the condition from the file operation to the particular index; and
selecting the subset of files based on results of said querying and without inspecting metadata of the set of files stored on the at least one storage device.
17. The device of claim 15 ,
wherein said tracking comprises scanning metadata of the set of files, wherein the metadata comprises the at least one attribute; and
wherein said querying comprises selecting the subset of files based on the subset of entries identified from said querying independently of said tracking or said scanning.
18. The device of claim 15 , wherein the one or more processors are further configured to:
scan the at least one file attribute of new or modified files from the set of files periodically with a first set of executable threads of the file system; and
set, by operation of the first set of executable threads, the particular index for an entry of a particular file in response to detecting the particular file in the set of files with the at least one attribute having one or more specified values.
19. The device of claim 18 , wherein the one or more processors are further configured to:
generate at least one executable thread from a second set of executable threads of the file syst that execute independently of the first set of executable threads in response to receiving the file operation; and
determine, by operation of the at least one executable thread of the second set of executable threads, the subset of files implicated by the file operation based on indexes in the database.
20. A non-transitory computer-readable medium, storing a set of processor-executable instructions, which, when executed by one or more processors, cause the one or more processors to:
implement a file system that stores a plurality of files to at least one storage device and that creates an entry in a database for each file of the plurality of files, wherein each entry comprises an identifier that identifies a storage location of a different file on the at least one storage device
track, by operation of the file system, at least one attribute from a set of the plurality of files based on the file system setting a particular index in each entry of a set of entries created in the database for the set of files;
receive a file operation at the file system;
query, by the file system, the database for a subset of the set of entries having the particular index set to one or more values that satisfy a condition of the file operation, wherein the condition is defined using the at least one attribute; and
perform, by the file system, the file operation against a subset of the set of files that is identified on the at least one storage device from the identifier associated with each entry of the subset of entries, and wherein performing the file operation comprises modifying the subset of files on the at least one storage device according to the file operation.