Systems and methods for file fingerprinting
A file fingerprint may be provided as a composite of multiple hashes of different portions of the file. The composite hash allows the fingerprinting process to be interrupted while still providing information about a likely hood of two files being identical.
1. A method comprising:
generating, based on metadata of a file, a read profile associated with the file, the read profile comprising a nominal number of hash passes, wherein the nominal number of hash passes is at least two;
generating a number of hashes according to the read profile, wherein the generated number of hashes is equal to the nominal number of hash passes; and
generating a file fingerprint associated with the file, the file fingerprint comprising the generated number of hashes.
2. The method of claim 1 , further comprising, after generating the number of hashes, determining to generate a next hash according to the read profile.
3. The method of claim 2 , wherein determining to generate the next hash is based on one or more of:
available computer resources;
input/output load; and
network performance.
4. The method of claim 1 , wherein the read profile specifies portions of the file to use to generate the number of hashes.
5. The method of claim 4 , wherein generating the number of hashes comprises generating, based on the read profile, a probe hash pass and one or more of a stripe pass hash and a block hash.
6. The method of claim 1 , further comprising, prior to generating the read profile, selecting the file from an ordered list of a plurality of files.
7. The method of claim 6 , wherein new files are added to the ordered list when the new files are accessed.
8. The method of claim 1 , wherein the file fingerprint further comprises a version identifier associated with the file fingerprint.
9. The method of claim 1 , further comprising storing the file fingerprint.
10. A system comprising:
a processor for executing instructions; and
a memory storing instructions which, when executed, cause the processor to:
generate, based on metadata of a file, a read profile associated with the file, the read profile comprising a nominal number of hash passes, wherein the nominal number of hash passes is at least two;
generate a number of hashes according to the read profile, wherein the generated number of hashes is equal to the nominal number of hash passes; and
generate a file fingerprint associated with the file, the file fingerprint comprising the generated number of hashes.
11. The system of claim 10 , wherein the instructions, when executed, further cause the processor to, after generating the number of hashes, determine to generate a next hash according to the read profile.
12. The system of claim 11 , wherein determining to generate the next hash is based on one or more of:
available computer resources;
input/output load; and
network performance.
13. The system of claim 10 , wherein the read profile specifies portions of the file to use to generate the number of hashes.
14. The system of claim 13 , wherein generating the number of hashes comprises generating, based on the read profile, a probe hash pass and one or more of a stripe pass hash and a block hash.
15. The system of claim 10 , wherein the instructions, when executed, further cause the processor to, prior to generating the read profile, select the file from an ordered list of a plurality of files.
16. The system of claim 15 , wherein new files are added to the ordered list when the new files are accessed.
17. The system of claim 10 , wherein the file fingerprint further comprises a version identifier associated with the file fingerprint.
18. The system of claim 10 , wherein the instructions, when executed, further cause the processor to store the generated file fingerprint.
19. A computer program product comprising:
a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code configured, when executed by a processor, to execute a method, the method comprising:
generating, based on metadata of a file, a read profile associated with the file, the read profile comprising a nominal number of hash passes, wherein the nominal number of hash passes is at least two;
generating a number of hashes according to the read profile, wherein the generated number of hashes is equal to the nominal number of hash passes; and
generating a file fingerprint associated with the file, the file fingerprint comprising the generated number of hashes.
20. The computer program product of claim 19 , further comprising, after generating the number of hashes, determining to generate a next hash according to the read profile.