Using a storage system to optimize and maintain the metadata associated with a plurality of small files
A size associated with a first content file is determined to be less than a threshold size. In response to determining that the size associated with the first content file is less than the threshold size, a combined metadata structure is updated at least in part by combining metadata of the first content file with metadata of a second content file in the combined metadata structure. A snapshot tree is updated to reference a first portion of the combined metadata structure corresponding to the first content file and to reference a second portion of the combined metadata structure corresponding to the second content file.
1. A method, comprising:
determining, by a storage system receiving a first content file from a storage device, that a storage size associated with the first content file is less than a threshold size, wherein metadata of the first content file is represented by a first file metadata structure;
in response to the determination that the storage size associated with the first content file is less than the threshold size updating, by the storage system, a combined metadata structure at least by:
determining, by the storage system and from a combined metadata structure reservation node of a tree data structure, an offset range associated with the combined metadata structure;
attempting, by the storage system, to successfully obtain a lock for the combined metadata structure;
in response to successfully obtaining the lock for the combined metadata structure, reserving, by the storage system, the offset range associated with the combined metadata structure; and
combining in the combined metadata structure, the metadata represented by the first file metadata structure with metadata of a second content file, the metadata of the second content file represented by a second file metadata structure, wherein a storage size associated with the second content file is less than the threshold size; and
updating, by the storage system, the tree data structure at least by updating a first reference that references a first root node of the first file metadata structure to reference a first portion of the combined metadata structure corresponding to the first content file and updating a second reference that references a second root node of the second file metadata structure to reference a second portion of the combined metadata structure corresponding to the second content file.
2. The method of claim 1 , wherein the combined metadata structure reservation node includes information associated with a plurality of combined metadata structures, wherein the plurality of combined metadata structures includes the combined metadata structure.
3. The method of claim 2 , wherein the information associated with the plurality of combined metadata structures includes offset ranges associated with each of the plurality of combined metadata structures, available offset ranges associated with each of the plurality of combined metadata structures, and/or a reserved offset range associated with the combined metadata structure.
4. The method of claim 2 , wherein a number of the plurality of combined metadata structures is a preset number.
5. The method of claim 2 , further comprising selecting, by the storage system, the combined metadata structure from a plurality of combined metadata structures.
6. The method of claim 5 , wherein selecting the combined metadata structure comprises selecting, by the storage system, the combined metadata structure based on an available offset range associated with the combined metadata structure.
7. The method of claim 5 , wherein selecting the combined metadata structure comprises selecting, by the storage system, the combined metadata structure based on an assignment policy.
8. The method of claim 1 , wherein reserving the offset range associated with the combined metadata structure includes updating, by the storage system, a data structure of the combined metadata structure reservation node.
9. The method of claim 1 , further comprising releasing, by the storage system, the lock for the combined metadata structure after combining in the combined metadata structure the metadata of the first content file with the metadata of the second content file.
10. The method of claim 1 , further comprising receiving, by the storage system, an updated version of the first content file.
11. The method of claim 10 , further comprising:
determining, by the storage system, a storage size associated with the updated version of the first content file is greater than or equal to the threshold size; and
removing, by the storage system, the metadata of the first content file from the combined metadata structure.
12. Non-transitory computer-readable media storing instructions that, when executed, cause processing circuitry to
determine, by receiving a first content file from a storage device, that a storage size associated with the first content file is less than a threshold size, wherein metadata of the first content file is represented by a first file metadata structure;
in response to the determination that the storage size associated with the first content file is less than the threshold size update a combined metadata structure at least by:
determining, from a combined metadata structure reservation node of a tree data structure, an offset range associated with the combined metadata structure;
attempting to successfully obtain a lock for the combined metadata structure;
in response to successfully obtaining the lock for the combined metadata structure, reserving the offset range associated with the combined metadata structure; and
combining in the combined metadata structure, the metadata represented by of the first file metadata structure with metadata of a second content file, the metadata of the second content file represented by a second file metadata structure, wherein a storage size associated with the second content file is less than the threshold size; and
update the tree data structure at least by updating a first reference that references a first root node of the first file metadata structure to reference a first portion of the combined metadata structure corresponding to the first content file and updating a second reference that references a second root node of the second file metadata structure to reference a second portion of the combined metadata structure corresponding to the second content file.
13. The non-transitory computer readable storage media of claim 12 , wherein the instructions, when executed, cause the processing circuitry to release the lock for the combined metadata structure after combining in the combined metadata structure the metadata of the first content file with the metadata of the second content file.
14. A system, comprising:
one or more processors configured to:
determine, by receiving a first content file from a storage device, that a storage size associated with the first content file is less than a threshold size, wherein metadata of the first content file is represented by a first file metadata structure;
in response to the determination that the storage size associated with the first content file is less than the threshold size update a combined metadata structure at least by:
determining, from a combined metadata structure reservation node of a tree data structure, an offset range associated with the combined metadata structure;
attempting to successfully obtain a lock for the combined metadata structure;
in response to successfully obtaining the lock for the combined metadata structure, reserving the offset range associated with the combined metadata structure; and
combining in the combined metadata structure, the metadata represented by the first file metadata structure with metadata of a second content file, the metadata of the second content file represented by a second file metadata structure, wherein a storage size associated with the second content file is less than the threshold size; and
update the tree data structure at least by updating a first reference that references a first root node of the first file metadata structure to reference a first portion of the combined metadata structure corresponding to the first content file and updating a second reference that references a second root node of the second file metadata structure to reference a second portion of the combined metadata structure corresponding to the second content file; and
a memory coupled to the one or more processors and configured to provide the one or more processors with instructions.
15. The method of claim 1 , further comprising deleting, by the storage system, the first file metadata structure and the second file metadata structure after combining in the combined metadata structure the metadata of the first content file with the metadata of the second content file.
16. The non-transitory computer readable storage media of claim 12 , wherein to reserve the offset range associated with the combined metadata structure the instructions, when executed, cause the processing circuitry to update a data structure of the combined metadata structure reservation node.
17. The non-transitory computer readable storage media of claim 12 , wherein the instructions, when executed, cause the processing circuitry to delete the first file metadata structure and the second file metadata structure after combining in the combined metadata structure the metadata of the first content file with the metadata of the second content file.
18. The system of claim 14 , wherein to reserve the offset range associated with the combined metadata structure the one or more processors are configured to update a data structure of the combined metadata structure reservation node.
19. The system of claim 14 , wherein the one or more processors are configured to delete the first file metadata structure and the second file metadata structure after combining in the combined metadata structure the metadata of the first content file with the metadata of the second content file.