Efficient similarity detection
A method for uploading a file to a server from multiple devices is disclosed. The server receives a request from a user to upload a file thereto. An upload client extracts file information from the file. The server generates a file signature for the file. The server accesses one or more existing file signatures for each of one or more existing files on the server. The server determines whether any of the one or more existing file signatures are similar to the file signature. Responsive to determining that there is an existing file signature that is similar to the file signature, the server accesses a first hash signature for the existing file. The upload client generates a second hash signature for the file. Responsive to determining that the first hash signature does not equal the second hash signature, the upload client uploads the file to the server.
1. A method for detecting similar files, the method comprising:
receiving, at a processor, a request from a user device to upload a file to a server;
extracting file information comprising at least a filename, a file size, and metadata from the file with an upload client, wherein the metadata includes information regarding creation of the file separate from content of the file;
generating, by the server, a file signature for the file based on at least the filename, the file size, and the metadata, wherein the file signature is different from a hash signature;
accessing one or more existing file signatures for each of one or more existing files stored on the server;
comparing the one or more existing file signatures to the file signature;
accessing a first hash signature for the existing file corresponding to the existing file signature;
upon determining that the file signature is within a predetermined deviation from one of the existing file signatures based on the comparison, generating a second hash signature for the file corresponding to the file signature; and
storing the file to the server responsive to determining that the first hash signature does not equal the second hash signature.
2. The method of claim 1 , wherein accessing the one or more existing file signatures for each of the one or more existing files on the server comprises:
communicating with a database hosting an account corresponding to the user; and
accessing the one or more existing file signatures associated with the account of the user.
3. The method of claim 1 , wherein accessing the first hash signature for the existing file corresponding to the existing file signature comprises:
communicating with a database hosting an account corresponding to the user;
accessing the first hash signature associated with the account of the user; and
transmitting the first hash signature to the upload client, wherein the second hash signature is generated for the file corresponding to the file signature with the upload client.
4. The method of claim 1 , further comprising:
receiving, by the processor, a request from the user device to upload another file to the server;
extracting file information comprising at least a filename, a file size, and metadata from the other file with the upload client;
generating, by the server, another file signature for the other file based on at least the filename, the file size, and the metadata from the other file;
accessing another one or more existing file signatures for each of another one or more existing files on the server;
comparing the other one or more existing file signatures to the other file signature;
accessing another first hash signature for the other existing file corresponding to the other existing file signature;
generating another second hash signature for the other file corresponding to the other file signature based on the comparison; and
following user directions for handling the file responsive to determining that the other first hash signature equals the other second hash signature.
5. The method of claim 4 , wherein following the user directions for handling the file comprises rejecting the file for upload.
6. The method of claim 4 , wherein following the user directions for handling the file comprises storing a new version of the file to the server.
7. The method of claim 1 , wherein comparing the one or more existing file signatures includes pixel analysis of the file.
8. A system for detecting similar files, the system comprising:
a processor; and
a memory having instructions stored thereon, which, when executed by the processor, performs an operation for detecting similar files, the operation, comprising:
receiving a request from a user device to upload a file to a server;
extracting file information comprising at least a filename, a file size, and metadata from the file with an upload client, wherein the metadata includes information regarding creation of the file separate from content of the file;
generating, by the server, a file signature for the file based on at least the filename, the file size, and the metadata, wherein the file signature is different from a hash signature;
accessing one or more existing file signatures for each of one or more existing files stored on the server;
comparing the one or more existing file signatures to the file signature;
accessing a first hash signature for the existing file corresponding to the existing file signature;
upon determining that the file signature is within a predetermined deviation from one of the existing file signatures based on the comparison, generating a second hash signature for the file corresponding to the file signature; and
storing the file to the server responsive to determining that the first hash signature does not equal the second hash signature.
9. The system of claim 8 , wherein accessing the one or more existing file signatures for each of the one or more existing files on the server comprises:
communicating with a database hosting an account corresponding to the user; and
accessing the one or more existing file signatures associated with the account of the user.
10. The system of claim 8 , wherein accessing the first hash signature for the existing file corresponding to the existing file signature comprises:
communicating with a database hosting an account corresponding to the user; accessing the first hash signature associated with the account of the user; and
transmitting the first hash signature to the upload client, wherein the second hash signature is generated for the file corresponding to the file signature with the upload client.
11. The system of claim 8 , further comprising:
receiving a request, by the processor, from the user device to upload another file to the server;
extracting file information comprising at least a filename, a file size, and metadata from the other file with the upload client;
generating, by the server, another file signature for the other file based on at least the filename, the file size, and the metadata from the other file;
accessing another one or more existing file signatures for each of another one or more existing files stored on the server;
comparing the other one or more existing file signatures are similar to the other file signature;
accessing another first hash signature for the other existing file corresponding to the other existing file signature;
generating another second hash signature for the other file corresponding to the other file signature based on the comparison; and
following user directions for handling the file responsive to determining that the other first hash signature equals the other second hash signature.
12. The system of claim 11 , wherein following the user directions for handling the file comprises rejecting the file for upload.
13. The system of claim 11 , wherein following the user directions for handling the file comprises storing a new version of the file to the server.
14. A non-transitory computer readable storage medium having instructions stored thereon, which, when executed by a processor, cause the processor to perform a method of detecting similar files, the method comprising:
receiving, by a processor, a request from a user device to upload a file to a server;
extracting file information comprising at least a filename, a file size, and metadata from the file with an upload client, wherein the metadata includes information regarding creation of the file separate from content of the file;
generating, by the server, a file signature for the file based on at least the filename, the file size, and the metadata, wherein the file signature is different from a hash signature;
accessing one or more existing file signatures for each of one or more existing files stored on the server;
comparing the one or more existing file signatures are similar to the file signature;
accessing a first hash signature for the existing file corresponding to the existing file signature;
upon determining that the file signature is within a predetermined deviation from one of the existing file signatures based on the comparison, generating a second hash signature for the file corresponding to the file signature; and
responsive to determining that the first hash signature does not equal the second hash signature, storing the file to the server.
15. The non-transitory computer readable storage medium of claim 14 , wherein accessing the one or more existing file signatures for each of the one or more existing files on the server comprises:
communicating with a database hosting an account corresponding to the user; and
accessing the one or more existing file signatures associated with the account of the user.
16. The non-transitory computer readable storage medium of claim 14 , wherein accessing the first hash signature for the existing file corresponding to the existing file signature comprises:
communicating with a database hosting an account corresponding to the user;
accessing the first hash signature associated with the account of the user; and
transmitting the first hash signature to the upload client, wherein the second hash signature is generated for the file corresponding to the file signature with the upload client.
17. The non-transitory computer readable storage medium of claim 14 , further comprising:
receiving, by the processor, a request from the user to upload another file to the server;
extracting file information comprising at least a filename, a file size, and metadata from the other file with the upload client;
generating, by the server, another file signature for the other file based on at least the filename, the file size, and the metadata from the other file;
accessing another one or more existing file signatures for each of another one or more existing files stored on the server;
comparing the other one or more existing file to the other file signature;
accessing another first hash signature for the other existing file corresponding to the other existing file signature;
generating another second hash signature for the other file corresponding to the other file signature based on the comparison; and
following user directions for handling the file responsive to determining that the other first hash signature equals the other second hash signature.
18. The non-transitory computer readable storage medium of claim 17 , wherein following the user directions for handling the file comprises rejecting the file for upload; and storing a new version of the file to the server.