Small file storage system
A small file storage system reduces the cost of storing and reading segmented data in a deduplication storage system. A data-blocking scheme is used to store small files instead of generating different levels of metadata as is done for larger files in deduplication systems. Among other advantages, the data blocking scheme reduces the cost of reading metadata segments from the content store, reduces the cost of index lookup, CM region read and decompression, and the cost of recalling and restoring small files from cloud storage, all of which contribute to better read performance. Other advantages include faster verification of small files, and improvements in small file data locality on disk with larger segment sizes. By eliminating the need to update metadata segments, embodiments of the small file storage system advantageously increase the efficiency of file updates, especially since updates to smaller files are more random in nature.
1. A computer-implemented method for managing small and very small files in an object storage, the method comprising:
determining whether a file is small in a file system having a data tree structure representing small files, the data tree structure including a data block map of any one or more segments comprising the small files, the data block map including any one of a variable block map, a fixed block map and a single block map;
determining whether the small file should be stored using any one of the variable block map, the fixed block map and the single block map without generating metadata associated with the small file;
generating the data block map for the small file, wherein generating the data block map for the small file includes:
generating the fixed block map for the small file in which the size of segments identified for the file range from 8 KB to 64 KB, including generating the fixed segment size for the data tree structure, generating fingerprints for each of the fixed segments comprising the small file,
generating the variable block map for small files in which the size of segments identified for the file range from 32 KB to 64 KB, including generating the variable segment size and fingerprints for each of the variable segments comprising the file, and
generating the single block map for small files where a size of the file is determined to be less than a threshold size for very small files of less than 10 KB, including generating the single data tree structure containing all of the data comprising the very small file; and
storing the small file in the object storage based on the generated data block map without generating the metadata.
2. The computer-implemented method of claim 1 , wherein the size of the file is small relative to sizes of other files in the file system when the size of the file is less than or equal to 100 KB.
3. The computer-implemented method of claim 1 , wherein the metadata includes a file name, a storage unit identifier, and reconstruction information for the small file using the segments.
4. The computer-implemented method of claim 1 , wherein the single block map includes small files in which the size of segments identified for the file is less than or equal to 10 KB.
5. The computer-implemented method of claim 1 , wherein the fixed data block map and the variable data block map do not contain segments of the metadata, and wherein the single data tree structure does not contain segments of the metadata and fingerprints identifying the segments of the small file.
6. A data processing system comprising:
a deduplication file system having a data tree structure representing small files, the data tree structure including a data block map of any one or more segments comprising the small files, the data block map including any one of a variable block map, a fixed block map and a single block map; and
a processor configured to store a file using the data block map including:
determining whether the file is small,
determining whether the small file should be stored using any one of the variable block map, the fixed block map and the single block map without generating metadata associated with the small file,
generating the data block map for the small file, wherein generating the data block map for the small file includes:
generating the fixed block map for small files in which a size of segments identified for the file range from 8 KB to 64 KB, including generating the fixed segment size for the data tree structure, generating fingerprints for each of the fixed segments comprising the small file,
generating the variable block map for small files in which the size of segments identified for the file range from 32 KB to 64 KB, including generating the variable segment size and fingerprints for each of the variable segments comprising the file, and
generating the single block map for small files where a size of the file is determined to be less than a threshold size for very small files of less than 10 KB, including generating the single data tree structure containing all of the data comprising the very small file,
and
storing the small file in an object storage based on the generated data block map without generating the metadata.
7. The data processing system of claim 6 , further comprising:
the processor configured to retrieve the file using the data block map including:
determining whether the file is a small file;
determining whether the small file was stored using any one of the variable block map, the fixed block map, and the single block map; and
retrieving the file in accordance with the any one of the variable block map, the fixed block map, and the single block map with which the small file was stored.
8. The data processing system of claim 6 , wherein the metadata includes a file name, a storage unit identifier, and reconstruction information for the small file using the segments.
9. A non-transitory computer-readable storage medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for managing small files in an object storage, the operations comprising:
determining whether a file is small in a file system having a data tree structure representing small files, the data tree structure including a data block map of any one or more segments comprising the small files, the data block map including any one of a variable block map, a fixed block map and a single block map;
determining whether the small file should be stored using any one of the variable block map, the fixed block map and the single block map without generating metadata associated with the small file;
generating the data block map for the small file, wherein generating the data block map for the small file includes:
generating the fixed block map for the small file in which the size of segments identified for the file range from 8 KB to 64 KB, including generating the fixed segment size for the data tree structure, generating fingerprints for each of the fixed segments comprising the small file,
generating the variable block map for small files in which the size of segments identified for the file range from 32 KB to 64 KB, including generating the variable segment size and fingerprints for each of the variable segments comprising the file, and
generating the single block map for small files where a size of the file is determined to be less than a threshold size for very small files of less than 10 KB, including generating the single data tree structure containing all of the data comprising the very small file; and
storing the small file in the object storage based on the generated data block map without generating the metadata.
10. The non-transitory computer-readable storage medium of claim 9 , wherein the size of the file is small relative to sizes of other files in the file system when the size is less than or equal to 100 KB.
11. The non-transitory computer-readable storage medium of claim 9 , wherein the metadata includes a file name, a storage unit identifier, and reconstruction information for the small file using the segments.
12. The non-transitory computer-readable storage medium of claim 9 , wherein the single block map includes small files in which the size of segments identified for the file is less than or equal to 10 KB.
13. The non-transitory computer-readable storage medium of claim 9 , wherein the fixed data block map and the variable data block map do not contain segments of the metadata, and wherein the single data tree structure does not contain segments of the metadata and fingerprints identifying the segments of the small file.