Efficient scanning for threat detection using in-doc markers
The disclosed technology teaches reducing threat detection processing, including recognizing that a file is an edited version of a previously processed file and retrieving, from an archive, metadata values, hashes for property groups and an entropy measure of the previously processed file. Also included is parsing the file into metadata values and property groups and calculating hashes of the property groups and entropy measure for the file. The method further includes applying similarity measures to compare the metadata values, the entropy measures, and the hashes on the property groups, for the edited version and the previously processed file. When any similarity measure or combination of similarity measures reaches a trigger, the technology teaches processing the file by using a threat detection module to detect malware. Property groups include core properties, application properties, document content and programming scripts for the edited version of the file and the previously processed file.
1. A computer-implemented method of reducing threat detection processing, including:
recognizing that a file is an edited version of a previously processed file;
retrieving, from an archive, metadata values, an entropy measure and hashes for property groups of the previously processed file;
parsing the file into metadata values and property groups, taking into account format of the file, and calculating the hashes of the property groups and an entropy measure for the file;
applying similarity measures to compare the metadata values, the entropy measures, and the hashes on property groups, for the edited version and the previously processed file; and
avoiding full threat scanning of the file to detect malware except when a similarity measure or a combination of the similarity measures reaches a scanning trigger.
2. The computer-implemented method of claim 1 , further including logging the edited version of the file for further processing when the similarity measure or the combination of the similarity measures reaches a logging trigger.
3. The computer-implemented method of claim 1 , further including performing a full threat scanning of the edited version of the file when the similarity measure or the combination of the similarity measures reaches the scanning trigger.
4. The computer-implemented method of claim 1 , wherein the hashes of the properties of the edited version and the previously processed file are locality preserving hashes.
5. The computer-implemented method of claim 1 , wherein the hashes of the properties of the edited version and the previously processed file are fuzzy hashes.
6. The computer-implemented method of claim 1 , wherein the metadata values include:
file size for the edited version of the file and for the previously processed file;
date for the edited version of the file, and date when the previously processed file was last scanned; and
file name and file extension for the edited version of the file and for the previously processed file.
7. The computer-implemented method of claim 1 , wherein the metadata values include file size for the edited version of the file and for the previously processed file.
8. The computer-implemented method of claim 1 , wherein the property groups of the edited version of the file and for the previously processed file include core properties, application properties, main document content and programming scripts.
9. The computer-implemented method of claim 1 , further including calculating and storing a hash for each property group for the edited version of the file: core properties, application properties, main document content and programming scripts.
10. The computer-implemented method of claim 1 , further including applying the similarity measures to more than one version of the previously processed file.
11. A tangible non-transitory computer readable storage media, including program instructions loaded into memory that, when executed on processors cause the processors to implement a method of reducing threat detection processing, the method including:
recognizing that a file is an edited version of a previously processed file;
retrieving, from an archive, metadata values, an entropy measure and hashes for property groups of the previously processed file;
parsing the file into metadata values and property groups, taking into account format of the file, and calculating the hashes of the property groups and an entropy measure for the file;
applying similarity measures to compare the metadata values, the entropy measures, and the hashes on property groups, for the edited version and the previously processed file; and
avoiding full threat scanning of the file to detect malware except when a similarity measure or a combination of the similarity measures reaches a scanning trigger.
12. The tangible non-transitory computer readable storage media of claim 11 , further including logging the edited version of the file for further processing when the similarity measure or the combination of the similarity measures reaches a logging trigger.
13. The tangible non-transitory computer readable storage media of claim 11 , further including performing a full threat scanning of the edited version of the file when the similarity measure or the combination of the similarity measures reaches the scanning trigger.
14. The tangible non-transitory computer readable storage media of claim 11 , wherein the hashes for the properties of the edited version and the previously processed file are locality preserving hashes.
15. The tangible non-transitory computer readable storage media of claim 11 , wherein the hashes for the properties of the edited version and the previously processed file are fuzzy hashes.
16. The tangible non-transitory computer readable storage media of claim 11 , wherein the metadata values include:
file size for the edited version of the file and for the previously processed file;
date for the edited version of the file, and date when the previously processed file was last scanned; and
file name and file extension for the edited version of the file and for the previously processed file.
17. The tangible non-transitory computer readable storage media of claim 11 , wherein the metadata values include file size for the edited version of the file and for the previously processed file.
18. The tangible non-transitory computer readable storage media of claim 11 , wherein the properties of the edited version of the file and for the previously processed file include core properties of the file, application properties, main document content and programming scripts.
19. The tangible non-transitory computer readable storage media of claim 11 , further including calculating and storing a hash for each property for the edited version of the file: core properties of the file, application properties, main document content and programming scripts.
20. A system for reducing threat detection processing, the system including a processor, memory coupled to the processor and computer instructions from the non-transitory computer readable storage media of claim 11 loaded into the memory.
21. The system of claim 20 , wherein the triggers for the similarity measures are configurable values.
22. The system of claim 20 , wherein the hashes for the properties of the edited version and the previously processed file are locality preserving hashes.
23. The system of claim 20 , further including applying the similarity measures to more than one version of the previously processed file.
24. The system of claim 20 , further including calculating and storing a hash for each property group for the edited version of the file: core properties, application properties, main document content and programming scripts.