Prefix fingerprint
A system and method for prefix fingerprints for a first file or a first data object. A prefix fingerprint comprises a plurality of hash values. The hash values of the prefix fingerprints are typically generated starting at the same offset within the file or data object, but are generated based on different data sizes. Later, a second file or second data object can be compared with the first file or first data object to determine if the second file or data object is a prefix of the first file or data object. A hash value is selected from the previously determined prefix fingerprint of the first file based on the size of the second file. A hash is generated for the second file using the same offset value and size as was used to generate the selected hash value from the prefix fingerprint. The hash values are then compared.
1. A method for comparing files, the method comprising:
comparing a first file with a second file, wherein a plurality of hash values are associated with a first prefix fingerprint for the first file, wherein the plurality of hash values were based on a plurality of data sections of the first file, the plurality of data sections having a same first beginning offset and different data sizes;
selecting a first hash value from the first prefix fingerprint based, at least in part, on a size of the second file;
generating a second hash value for the second file, the second hash value based on a data section of the second file, the data section of the second file having a second beginning offset equal to the first beginning offset and the same data size used to generate the first hash value in the first prefix fingerprint;
a fingerprint dataset configured to store the first prefix fingerprint communicably coupled with a computing device, wherein the fingerprint dataset has a file identifier (ID) and the first prefix fingerprint has one or more hash values covering different sizes of data in the first file,
wherein the fingerprint dataset is a hash table in order to determine whether the second file is a prefix of another file in a constant O(1) time;
comparing the first hash value to the second hash value; and
determining, based on the comparison, whether the second file is a prefix of the first file.
2. The method of claim 1 wherein the first beginning offset is zero.
3. The method of claim 1 further comprising indicating that the second file is an incomplete download of the first file in response to determining that the first hash value equals the second hash value.
4. The method of claim 1 wherein data sizes for successive hash values in the first prefix fingerprint increase in size.
5. A non-transitory computer readable storage medium having a program stored therein, the program causing a computer to execute:
receiving a first file;
generating a prefix fingerprint for the first file;
wherein the prefix fingerprint comprises one or more hash values calculated from a beginning offset value in the first file;
comparing the first file with a second file wherein a plurality of hash values are associate with the first prefix fingerprint for the first file, wherein the plurality of hash values are based on a plurality of data sections of the first file, the plurality of data sections having a same first beginning offset and different data sizes;
selecting a first hash value from the prefix fingerprint based, at least in part, on a size of the second file;
generating a second hash value for the second file, the second hash value based on a data section of the second file, the data section of the second file having a second beginning offset equal to the first beginning offset and the same data size used to generate the first hash value in the prefix fingerprint;
a fingerprint dataset storing the first prefix fingerprint communicably coupled with a computing device, wherein the fingerprint dataset has a file identifier (ID) and the first prefix fingerprint has one or more hash values covering different sizes of data in the first file,
wherein the fingerprint dataset is a hash table in order to determine whether the second file is a prefix of another file in a constant O(1) time;
comparing the first hash value to the second hash value; and
determining, based on the comparison, whether the second file is a prefix of the first file.