System and method for performing an antivirus scan using file level deduplication
Aspects of the disclosure describe methods and systems for performing an antivirus scan using file level deduplication. In an exemplary aspect, prior to performing an antivirus scan on files stored on at least two storage devices, a deduplication module calculates a respective hash for each respective file stored on the storage devices. The deduplication module identifies a first file stored the storage devices and determines whether at least one other copy of the first file exists on the storage devices. In response to determining that another copy exists, the deduplication module stores the first file in a shared database, replaces all copies of the first file on the storage devices with a link to the first file in the shared database, and performs the antivirus scan on (1) the first file in the shared database and (2) the files stored on the storage devices.
1. A method for performing an antivirus scan using file level deduplication, the method comprising:
prior to performing an antivirus scan on files stored on a first storage device and a second storage device:
calculating a respective hash for each respective file stored on the first storage device and the second storage device;
identifying a first file stored on the first storage device;
determining whether at least one other copy of the first file exists on either the first storage device or the second storage device based on the calculated respective hashes;
in response to determining that at least one other copy of the first file exists on either the first storage device or the second storage device:
storing the first file in a shared database; and
replacing all copies of the first file on the first storage device and the second storage device each with a link to the first file in the shared database;
detecting an operation being performed on the first file that has been replaced with the link;
retrieving the first file using the link;
performing the operation on the retrieved first file;
replacing the retrieved first file with the link; and
performing the antivirus scan on (1) the first file in the shared database and (2) the files stored on the first storage device and the second storage device.
2. The method of claim 1 , further comprising:
identifying a second file stored on the first storage device;
in response to determining, based on the calculated hashes, that at least one other copy of the second file exists on either the first storage device or the second storage device:
storing the second file in the shared database; and
replacing all copies of the second file on the first storage device and the second storage device each with a link to the second file in the shared database.
3. The method of claim 1 , wherein the first storage device and the second storage device are comprised in one computing device.
4. The method of claim 1 , wherein the first storage device is comprised in a first computing device and the second storage device is comprised in a second computing device, wherein the first computing device, the second computing device, and the shared database are all connected via a communication network.
5. The method of claim 1 , further comprising:
in response to determining that the at least one other copy of the first file is stored on the first storage device, deleting the at least one other copy from the first storage device.
6. The method of claim 1 , further comprising:
creating a record of the first file in the shared database, wherein the record comprises a hash of the first file;
subsequent to performing the antivirus scan, marking in the record of the first file an indication that the first file has been scanned.
7. The method of claim 6 , further comprising:
subsequent to the first file in the shared database being accessed via the link, determining whether the hash of the first file has changed; and
in response to determining that the hash has changed, rescanning the first file.
8. The method of claim 1 , further comprising:
in response to determining that at least one other copy of the first file does not exist on either the first storage device or the second storage device, storing a hash of the first file in the shared database along with a status of antivirus scan for the file.
9. The method of claim 1 , further comprising:
creating a new version of the first file in the shared database; and
replacing the retrieved first file with a new link to the new version only at a storage device where the operation was performed.
10. A system for performing an antivirus scan using file level deduplication, the system comprising:
a hardware processor configured to:
prior to performing an antivirus scan on files stored on a first storage device and a second storage device:
calculate a respective hash for each respective file stored on the first storage device and the second storage device;
identify a first file stored on the first storage device;
determine whether at least one other copy of the first file exists on either the first storage device or the second storage device based on the calculated respective hashes;
in response to determining that at least one other copy of the first file exists on either the first storage device or the second storage device:
store the first file in a shared database; and
replace all copies of the first file on the first storage device and the second storage device each with a link to the first file in the shared database;
detect an operation being performed on the first file that has been replaced with the link;
retrieve the first file using the link;
perform the operation on the retrieved first file;
replace the retrieved first file with the link; and
perform the antivirus scan on (1) the first file in the shared database and (2) the files stored on the first storage device and the second storage device.
11. The system of claim 10 , wherein the hardware processor is further configured to:
identify a second file stored on the first storage device;
in response to determining, based on the calculated hashes, that at least one other copy of the second file exists on either the first storage device or the second storage device:
store the second file in the shared database; and
replace all copies of the second file on the first storage device and the second storage device each with a link to the second file in the shared database.
12. The system of claim 10 , wherein the first storage device and the second storage device are comprised in one computing device.
13. The system of claim 10 , wherein the first storage device is comprised in a first computing device and the second storage device is comprised in a second computing device, wherein the first computing device, the second computing device, and the shared database are all connected via a communication network.
14. The system of claim 10 , wherein the hardware processor is further configured to:
in response to determining that the at least one other copy of the first file is stored on the first storage device, delete the at least one other copy from the first storage device.
15. The system of claim 10 , wherein the hardware processor is further configured to:
create a record of the first file in the shared database, wherein the record comprises a hash of the first file;
subsequent to performing the antivirus scan, mark in the record of the first file an indication that the first file has been scanned.
16. The system of claim 15 , wherein the hardware processor is further configured to:
subsequent to the first file in the shared database being accessed via the link, determine whether the hash of the first file has changed; and
in response to determining that the hash has changed, rescan the first file.
17. The system of claim 10 , wherein the hardware processor is further configured to:
create a new version of the first file in the shared database; and
replace the retrieved first file with a new link to the new version only at a storage device where the operation was performed.
18. A non-transitory computer readable medium storing thereon computer executable instructions for performing an antivirus scan using file level deduplication, including instructions for:
prior to performing an antivirus scan on files stored on a first storage device and a second storage device:
calculating a respective hash for each respective file stored on the first storage device and the second storage device;
identifying a first file stored on the first storage device;
determining whether at least one other copy of the first file exists on either the first storage device or the second storage device based on the calculated respective hashes;
in response to determining that at least one other copy of the first file exists on either the first storage device or the second storage device:
storing the first file in a shared database; and
replacing all copies of the first file on the first storage device and the second storage device each with a link to the first file in the shared database;
detecting an operation being performed on the first file that has been replaced with the link;
retrieving the first file using the link;
performing the operation on the retrieved first file;
replacing the retrieved first file with the link; and
performing the antivirus scan on (1) the first file in the shared database and (2) the files stored on the first storage device and the second storage device.