File system enabling fast purges and file access
Some embodiments provide a file system for purging content based on a single traversal of the file system that identifies the directory containing the content without performing a secondary traversal within the directory to target the operation to only the file that are associated with content such that other files contained in the directory are unaffected. The file system supplements traditional directory structures with file-level directories. Each file-level directory is created to contain a root file associated with particular content, different variants of the particular content, and supporting files. Consequently, the file system can complete an operation targeting particular content by performing that operation on the file-level directory containing the particular content, thereby eliminating the need to conduct a file-by-file traversal of the containing directory as a prerequisite to identifying the files associated with the particular content and performing the operation on the files individually.
1. A computer-implemented method comprising:
storing a plurality of files identified by a plurality of filenames to storage of a particular machine;
receiving a set of purge commands at the particular machine, each purge command of the set of purge commands identifying a different file from a subset of the plurality of files to purge from storage;
analyzing commonality in filenames for each file in the subset of files with a processor of the particular machine;
generating based on said commonality, one regular expression inclusive of the filename of each file in the subset of files and exclusive of the filename of each file in the plurality of files not also in the subset of files;
issuing the one regular expression in place of the set of purge commands to said storage; and
purging the subset of files from said storage based on the one regular expression selecting each file of the subset of files in a single traversal of a file system directory storing the subset of files, wherein each command of the set of purge commands executes with a separate traversal of the file system directory.
2. The computer-implemented method of claim 1 , wherein receiving the set of purge commands comprises grouping individual purge commands during a period of high demand for subsequent batch purging during a period of low demand.
3. The computer-implemented method of claim 2 , wherein said issuing occurs at the period of low demand.
4. The computer-implemented method of claim 1 , wherein said regular expression comprises a pattern matching to the filename of each file in the subset of files and not matching to the filename of each file in the plurality of files and not in the subset of files.
5. The computer-implemented method of claim 1 , wherein the plurality of files are stored to a common file system directory, the method further comprising grouping to the set of purge commands, any purge command directed to purging any file of the plurality of files in the common file system directory.
6. The computer-implemented method of claim 1 , wherein said purging comprises deleting during said single traversal of the file system directory, each file of the subset of files based on a filename matching the one regular expression.
7. The computer-implemented method of claim 1 further comprising grouping individual purge commands to the set of purge commands during an interval and issuing the one regular expression at the end of the interval.
8. The computer-implemented method of claim 1 , wherein the one regular expression comprises common alphanumeric characters from the filenames of the subset of files and at least one symbol that is not in the filenames of the subset of files.
9. A computer-implemented method for performing at least a two-part delayed purging of files from a file system, the method comprising:
receiving a purge command to delete at least one file stored in a first directory path of the file system of a particular machine;
determining load on the file system exceeds a specified amount;
changing the first directory path to a different second directory path without physical deletion of the at least one file from the file system in response to the load on the file system exceeding the specified amount and further in response to said receiving the purge command to delete the at least one file stored in the first directory path;
identifying when the load on the file system does not exceed the specified amount; and
purging the second directory path from the file system of the particular machine in response to the load on the file system not exceeding the specified amount and further in response said receiving the purge command to delete the at least one file stored in the first directory path, wherein said purging comprises physical deletion of the at least one file from the file system.
10. The computer-implemented method of claim 9 , wherein changing the first directory path to the different second directory path comprises renaming a directory under which the at least one file is stored.
11. The computer-implemented method of claim 9 , wherein said purging comprises performing a single traversal of the file system to delete a directory in which the at least one file is stored without performing a second traversal within the directory to identify and delete the at least one file.
12. The computer-implemented method of claim 9 further comprising storing the at least one file to the first directory path, wherein said storing comprises generating in the first directory path, a subdirectory based on a name of the at least one file and storing the at least one file in said subdirectory.
13. The computer-implemented method of claim 9 further comprising monitoring the load on the file system, wherein said monitoring comprises observing resource utilization on the particular machine.
14. A computer-implemented method comprising:
receiving a storage operation identifying within a file system, a directory path as a storage location for a particular file with a particular filename;
separating storage of the particular file from other files in the directory path by creating under the directory path, a file-level directory derived from the particular filename and storing the particular file in the file-level directory rather than the directory path specified by the storage operation;
receiving an original purge operation comprising the directory path and the particular filename, said original purge operation requesting deletion of the particular file from the directory path of the file system;
producing a modified purge operation from the original purge operation by appending the file-level directory to the directory path; and
issuing to the file system, the modified purge operation with the appended file-level directory in place of the original purge operation.
15. The computer-implemented method of claim 14 further comprising performing the modified purge operation by deleting the file-level directory without traversing the file-level directory and individually deleting the particular file therein.
16. The computer-implemented method of claim 14 , wherein the particular file is a first variant of first content, the method further comprising receiving a storage operation identifying within the file system, the directory path as a storage location for a different second file with a different second filename.
17. The computer-implemented method of claim 16 further comprising creating under the directory path, a different second file-level directory derived from the second filename and storing the second file in the second file-level directory rather than the directory path or the file-level directory of the particular file when the second file is a variant of different second content.
18. The computer-implemented method of claim 17 further comprising storing the second file to the file-level directory in which the particular file is stored when the second file is a different second variant of the first content.