Storage system garbage collection and defragmentation
Metadata of each file of a group of files of a storage and chunk file metadata are analyzed to identify one or more file segment data chunks that are not referenced by the group of files of the storage. Fragmented chunk files to be combined together are identified based at least in part on the one or more identified file segment data chunks. The chunk file metadata is updated with an update that concurrently reflects the removal of at least a portion of the one or more file segment data chunks that are not referenced by the group of files and the combination of the identified fragmented chunk files.
1. A method, comprising:
scanning a plurality of tree data structures corresponding to a plurality of files of a storage to identify a corresponding number of references associated with file segment data chunks included in the plurality of files;
analyzing the plurality of scanned tree data structures corresponding to the plurality of files and chunk file metadata, wherein analyzing the plurality of scanned tree data structures corresponding to the plurality of files and the chunk file metadata comprises:
identifying one or more file segment data chunks that are not referenced by the plurality of files of the storage based on an analysis of the corresponding scanned tree data structure associated with each file of the plurality of files, wherein the one or more file segment data chunk are identified as the one or more file segment data chunks that are not referenced by the plurality of files of the storage based on a corresponding reference count; and
identifying fragmented chunk files to be combined together based at least in part on the one or more identified file segment data chunks; and
combining file segment data chunks that are referenced by a group of files of the identified fragmented chunk files into one or more combined chunk files.
2. The method of claim 1 , further comprising generating one of the one or more combined chunk files using at least two of the identified fragmented chunk files.
3. The method of claim 2 , further comprising deleting the at least two of the identified fragmented chunk files after the one of the one or more combined chunk files are generated.
4. The method of claim 1 , further comprising grouping the identified fragmented chunk files into a plurality of groups based on a corresponding file associated with the identified fragmented chunk files.
5. The method of claim 4 , wherein for a group of the plurality of groups, the method further comprises sorting the identified fragmented chunk files included in the group based on a corresponding chunk file score.
6. The method of claim 5 , wherein the corresponding chunk file score indicates an amount of storage to be reclaimed in the event a fragmented chunk file is used to generate a combined chunk file.
7. The method of claim 1 , wherein the fragmented chunk files are identified based on an amount of unreferenced data chunks included in a chunk file.
8. The method of claim 1 , wherein identifying one or more file segment data chunks that are not referenced by the plurality of files of the storage, traversing the corresponding tree data structure associated with each file of the plurality of files.
9. The method of claim 8 , further comprising determining a chunk identifier and a chunk file identifier based on a traversal of the corresponding tree data structure.
10. The method of claim 9 , wherein the chunk identifier is associated with a file segment data chunk, wherein the file segment data chunk has a corresponding reference count.
11. The method of claim 10 , wherein a fragmented chunk file of the fragmented chunk files includes at least one data chunk with a corresponding reference count being zero.
12. The method of claim 1 , further comprising updating chunk metadata based on the one or more combined chunk files.
13. A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
scanning a plurality of tree data structures corresponding to a plurality of files of a storage to identify a corresponding number of references associated with file segment data chunks included in the plurality of files;
analyzing the plurality of scanned tree data structures corresponding to the plurality of files and chunk file metadata, wherein analyzing the plurality of scanned tree data structures corresponding to the plurality of files and the chunk file metadata comprises:
identifying one or more file segment data chunks that are not referenced by the plurality of files of the storage based on an analysis of the corresponding scanned tree data structure associated with each file of the plurality of files, wherein the one or more file segment data chunk are identified as the one or more file segment data chunks that are not referenced by the plurality of files of the storage based on a corresponding reference count; and
identifying fragmented chunk files to be combined together based at least in part on the one or more identified file segment data chunks; and
combining file segment data chunks that are referenced by a group of files of the identified fragmented chunk files into one or more combined chunk files.
14. The computer program product of claim 13 , further comprising instructions for generating one of the one or more combined chunk files based on the identified fragmented chunk files.
15. The computer program product of claim 14 , wherein the fragmented chunk files to be combined together are identified based on a corresponding chunk file score.
16. The computer program product of claim 15 , wherein the corresponding chunk file score indicates an amount or percentage that a chunk file includes unreferenced data chunks.
17. The computer program product of claim 15 , wherein the corresponding chunk file score indicates an amount or percentage that a chunk file includes unreferenced data chunks.
18. The computer program product of claim 13 , further comprising instructions for:
grouping the identified fragmented chunk files based on a file associated with the identified fragmented chunk file; and
for each group, sorting the identified fragmented chunk files based on their corresponding chunk file scores.
19. The computer program product of claim 14 , further comprising updating chunk metadata based on the combined chunks files.
20. A system, comprising:
a processor configured to:
scan a plurality of tree data structures corresponding to a plurality of files of a storage to identify a corresponding number of references associated with file segment data chunks included in the plurality of files;
analyze the plurality of scanned tree data structures corresponding to the plurality of files and chunk file metadata, wherein to analyze the plurality of scanned tree data structures corresponding to the plurality of files and the chunk file metadata, the processor is configured to:
identify one or more file segment data chunks that are not referenced by the plurality of files of the storage based on an analysis of the corresponding scanned tree data structure associated with each file of the plurality of files, wherein the one or more file segment data chunk are identified as the one or more file segment data chunks that are not referenced by the plurality of files of the storage based on a corresponding reference count; and
identify fragmented chunk files to be combined together based at least in part on the one or more identified file segment data chunks; and
combine file segment data chunks that are referenced by a group of files of the identified fragmented chunk files into one or more combined chunk files; and
a memory coupled to the processor and configured to provide the processor with instructions.