System and method for filesystem deduplication using variable length sharing
Embodiments of the present invention are directed to a method and system for filesystem deduplication that uses both small fingerprint granularity and variable length sharing techniques. The method includes accessing, within an electronic system, a plurality of files in a primary storage filesystem and determining a plurality of fingerprints for the plurality of files. Each respective fingerprint may correspond to a respective portion of a respective file of the plurality of files. The method further includes determining a plurality of portions of the plurality of files where each of the plurality of portions has the same corresponding fingerprint and accessing a list comprising a plurality of portions of files previously deduplicated. A portion of a file of the plurality of files not present in the list may then be deduplicated. Consecutive portions of variables lengths having the same corresponding fingerprints may also be deduplicated.
1. A method for filesystem deduplication comprising:
accessing, within an electronic system, a plurality of files in a primary storage filesystem;
determining a plurality of fingerprints for said plurality of files, wherein each respective fingerprint corresponds to a respective portion of a respective file of said plurality of files;
matching a first portion of a first file of said plurality of files with a first portion of a second file of said plurality of files, wherein said matched first portions have the same corresponding fingerprint;
determining a largest possible match, wherein said largest possible match comprises said matched first portions and all portions stored consecutive to said matched first portions in said first file and said second file that have the same corresponding fingerprint;
accessing a list comprising a plurality of portions of files previously deduplicated; and
deduplicating a second portion of a said first file not present in said list, wherein said second portion comprises at least said first portion of said first file and all said portions stored consecutive to said first portion in said first file comprising said largest possible match.
2. The method of claim 1 further comprising:
adding said second portion of said first file of said plurality of files to said list.
3. The method of claim 1 further comprising:
adding consecutive portions of said plurality of files having the same fingerprint to said list.
4. The method of claim 1 wherein said deduplicating is performed by via a kernel and said deduplicating a second portion of said first file not present in said list is performed by said kernel.
5. The method of claim 1 wherein said deduplicating further comprises setting a metadata pointer for a portion of a file.
6. The method of claim 1 wherein said list further comprises deleted files.
7. The method of claim 1 wherein said list further comprises truncated files.
8. The method of claim 1 further comprising:
verifying a fingerprint corresponding to said second portion of said first file prior to said deduplicating.
9. A non-transitory computer readable storage medium having stored thereon, computer executable instructions that, if executed by a computer system cause the computer system to perform a method for filesystem deduplication comprising:
accessing, within an electronic system, a plurality of files in a primary storage filesystem;
determining a plurality of fingerprints for said plurality of files, wherein each respective fingerprint corresponds to a respective portion of a respective file of said plurality of files;
matching a first portion of a first file of said plurality of files with a first portion of a second file of said plurality of files, wherein said matched first portions have the same corresponding fingerprint;
determining a largest possible match, wherein said largest possible match comprises said matched first portions and all portions stored consecutive to said matched first portions in said first file and said second file that have the same corresponding fingerprint;
accessing a list comprising a plurality of portions of files previously deduplicated; and
deduplicating a second portion of a said first file not present in said list, wherein said second portion comprises at least said first portion of said first file and all said portions stored consecutive to said first portion in said first file comprising said largest possible match.
10. The non-transitory computer readable storage medium of claim 9 , wherein said method further comprises:
adding said second portion of said first file of said plurality of files to said list.
11. The non-transitory computer readable storage medium of claim 9 , wherein said method further comprises:
adding consecutive portions of said plurality of files having the same fingerprint to said list.
12. The non-transitory computer readable storage medium of claim 9 , wherein said deduplicating is performed by via a kernel and said deduplicating a second portion of said first file not present in said list is performed by said kernel.
13. The non-transitory computer readable storage medium of claim 9 , wherein said deduplicating further comprises setting a metadata pointer for a portion of a file.
14. The non-transitory computer readable storage medium of claim 9 , wherein said list further comprises deleted files and truncated files.
15. The non-transitory computer readable storage medium of claim 9 , wherein said method further comprises:
verifying a fingerprint corresponding to said second portion of said first file prior to said deduplicating.
16. A storage system, comprising:
a computer system comprising one or more processors coupled to a computer readable storage media and configurable to execute computer readable code stored on the computer readable storage media which causes the computer system to implement a method comprising:
accessing, within an electronic system, a plurality of files in a primary storage filesystem;
determining a plurality of fingerprints for said plurality of files, wherein each respective fingerprint corresponds to a respective portion of a respective file of said plurality of files;
matching a first portion of a first file of said plurality of files with a first portion of a second file of said plurality of files, wherein said matched first portions have the same corresponding fingerprint;
determining a largest possible match, wherein said largest possible match comprises said matched first portions and all portions stored consecutive to said matched first portions in said first file and said second file that have the same corresponding fingerprint;
accessing a list comprising a plurality of portions of files previously deduplicated; and
deduplicating a second portion of a said first file not present in said list, wherein said second portion comprises at least said first portion of said first file and all said portions stored consecutive to said first portion in said first file comprising said largest possible match.
17. The storage system of claim 16 , wherein said method further comprises:
adding said second portion of said first file of said plurality of files to said list.
18. The storage system of claim 16 wherein said method further comprises:
adding consecutive portions of said plurality of files having the same fingerprint to said list.
19. The storage system of claim 16 wherein said method further comprises:
verifying a fingerprint corresponding to said second portion of said first file prior to said deduplicating.
20. The storage system of claim 18 wherein said deduplicating is performed by via a kernel and said deduplicating a second portion of said first file not present in said list is performed by said kernel.