Fuzzy whitelisting anti-malware systems and methods
In some embodiments, an anti-malware system accounts for benign differences between non-malicious data objects, such as differences introduced by compilers and other polymorphisms. A target object is separated into a multitude of code blocks, and a hash is calculated for each code block. The obtained set of target hashes is then compared against a database of hashes corresponding to code blocks extracted from whitelisted objects. A target object may be labeled as whitelisted (trusted, non-malicious) if it has a substantial number of hashes in common with a whitelisted object. Objects which are slightly different from known whitelisted objects may still receive whitelisting status. By allowing a certain degree of mismatch between the sets of hashes of distinct objects, some embodiments of the present invention increase the efficiency of whitelisting without an unacceptable decrease in safety.
1. A method comprising:
performing at a client computer system an initial malware scan of a plurality of target objects of the client computer system; and
in response to a tentative determination by the initial malware scan that the target object is suspected of being malicious:
generating at the client computer system a plurality of target hashes of the target object, each target hash representing a distinct code block of the target object, each distinct code block consisting of a sequence of processor instructions of the target object;
sending from the client computer system the plurality of target hashes to a server computer system connected to the client computer system over a wide area network; and
receiving at the client computer system from the server computer system a server-side indicator of whether the target object is malicious, wherein the server-side indicator is generated by the server computer system by:
for at least a target hash of the plurality of target hashes, retrieving a plurality of reference hashes of a reference object, the reference object selected from a set of whitelisted objects according to the target hash, and when the plurality of target hashes is not identical to the plurality of reference hashes, determining a similarity score according to a count of hashes common to both the plurality of target hashes and the plurality of reference hashes; and
when the similarity score exceeds a predetermined threshold, designating the target object as non-malicious.
2. The method of claim 1 , wherein generating the server-side indicator by the computer server system comprises:
when the similarity score does not exceed the predetermined threshold, generating a filtered set of target hashes of the target object by filtering all target hashes which appear in a database of clean hashes out of the plurality of target hashes of the target object; and
comparing the filtered set of target hashes to a database of malware-identifying hashes specific to malware.
3. The method of claim 1 , wherein generating the server-side indicator by the computer server system comprises:
when the similarity score does not exceed the predetermined threshold, generating a filtered set of target hashes of the target object by filtering all target hashes which appear in a database of clean hashes out of the plurality of target hashes of the target object; and
comparing the filtered set of target hashes to a database of outbreak-detection hashes specific to unknown objects reported within a predetermined recent period by a plurality of distinct client computer systems connected to the server computer system.
4. A method comprising:
receiving at a server computer system a plurality of target hashes of a target object of a client computer system connected to the server computer system over a wide area network, wherein the plurality of target hashes are generated at the client computer system in response to a tentative determination by the client computer system that the target object is suspected of being malicious, the tentative determination resulting from an initial malware scan of a plurality of target objects of the client computer system;
generating at the server computer system a server-side indicator of whether the target object is malicious by:
for at least a target hash of the plurality of target hashes, retrieving a plurality of reference hashes of a reference object, the reference object selected from a set of whitelisted objects according to the target hash, and when the plurality of target hashes is not identical to the plurality of reference hashes, determining a similarity score according to a count of hashes common to both the plurality of target hashes and the plurality of reference hashes, and
when the similarity score exceeds a predetermined threshold, designating the target object as non-malicious; and
sending to the client computer system the server-side indicator of whether the target object is malicious.
5. The method of claim 4 , wherein generating the server-side indicator by the computer server system comprises:
when the similarity score does not exceed the predetermined threshold, generating a filtered set of target hashes of the target object by filtering all target hashes which appear in a database of clean hashes out of the plurality of target hashes of the target object; and
comparing the filtered set of target hashes to a database of malware-identifying hashes specific to malware.
6. The method of claim 4 , wherein generating the server-side indicator by the computer server system comprises:
when the similarity score does not exceed the predetermined threshold, generating a filtered set of target hashes of the target object by filtering all target hashes which appear in a database of clean hashes out of the plurality of target hashes of the target object; and
comparing the filtered set of target hashes to a database of outbreak-detection hashes specific to unknown objects reported within a predetermined recent period by a plurality of distinct client computer systems connected to the server computer system.
7. A method comprising:
receiving at a server computer system a plurality of target hashes of a target object, each target hash representing a distinct code block of the target object, each distinct code block consisting of a sequence of processor instructions of the target object;
for at least a target hash of the plurality of target hashes, employing the server computer system to:
retrieve a plurality of reference hashes of a reference object, the reference object selected from a set of whitelisted objects according to the target hash, and
when the plurality of target hashes is not identical to the plurality of reference hashes, determine a similarity score according to a count of hashes common to both the plurality of target hashes and the plurality of reference hashes; and
when the similarity score exceeds a predetermined threshold, employing the server computer system to label the target object as non-malicious.
8. The method of claim 7 , wherein the target hash comprises a hash of an opcode pattern, the opcode pattern comprising a sequence of instruction indicators, each instruction indicator indicative of a processor instruction of the distinct code block.
9. The method of claim 7 , wherein the sequence of processor instructions consists of between 5 and 50 consecutive processor instructions.
10. The method of claim 9 , wherein the sequence of processor instructions consists of between 5 and 15 consecutive processor instructions.
11. The method of claim 7 , wherein the sequence of processor instructions starts with a CALL instruction.
12. The method of claim 7 , further comprising:
performing a code normalization procedure on the target object to produce a normalized target object, and wherein each distinct code block consists of a sequence of computer instructions of the normalized target object; and
applying a hash function to the distinct code block to produce the target hash.
13. The method of claim 7 , wherein the similarity score is determined as a function of:
C /max( N T ,N R )
wherein C denotes the count of hashes common to both the plurality of target hashes and the plurality of reference hashes, while N T and N R denote the cardinality of the plurality of target hashes and the cardinality of the plurality of reference hashes, respectively.
14. The method of claim 7 , wherein the similarity score is determined as a function of:
C /( N T +N R )
wherein C denotes the count of hashes common to both the plurality of target hashes and the plurality of reference hashes, while N T and N R denote the cardinality of the plurality of target hashes and the cardinality of the plurality of reference hashes, respectively.
15. The method of claim 7 , wherein the similarity score is determined as a function of:
C/N T +C/N R
wherein C denotes the count of hashes common to both the plurality of target hashes and the plurality of reference hashes, while N T and N R denote the cardinality of the plurality of target hashes and the cardinality of the plurality of reference hashes, respectively.
16. The method of claim 7 , wherein the target object comprises a computer file.
17. The method of claim 7 , wherein the target object comprises a computer process.
18. A computer system comprising at least a processor programmed to:
receive a plurality of target hashes, each target hash representing a distinct code block of a target object, each distinct code block consisting of a sequence of processor instructions of the target object;
for at least a target hash of the plurality of target hashes:
retrieve a plurality of reference hashes of a reference object, the reference object selected from a set of whitelisted objects according to the target hash, and
when the plurality of target hashes is not identical to the plurality of reference hashes, determine a similarity score according to a count of hashes common to both the plurality of target hashes and the plurality of reference hashes; and
when the similarity score exceeds a predetermined threshold, label the target object as non-malicious.
19. The system of claim 18 , wherein the target hash comprises a hash of an opcode pattern, the opcode pattern comprising a sequence of instruction indicators, each instruction indicator indicative of a processor instruction of the distinct code block.
20. The system of claim 18 , wherein the sequence of processor instructions consists of between 5 and 50 consecutive processor instructions.
21. The system of claim 20 , wherein the sequence of processor instructions consists of between 5 and 15 consecutive processor instructions.
22. The system of claim 18 , wherein the sequence of processor instructions starts with a CALL instruction.
23. The system of claim 18 , wherein the processor is further programmed to:
perform a code normalization procedure on the target object to produce a normalized target object, wherein each distinct code block consists of a sequence of computer instructions of the normalized target object; and
apply a hash function to the distinct code block to produce the target hash.
24. The system of claim 18 , wherein the similarity score is determined as a function of:
C /max( N T ,N R )
wherein C denotes the count of hashes common to both the plurality of target hashes and the plurality of reference hashes, while N T and N R denote the cardinality of the plurality of target hashes and the cardinality of the plurality of reference hashes, respectively.
25. The system of claim 18 , wherein the similarity score is determined as a function of:
C /( N T +N R )
wherein C denotes the count of hashes common to both the plurality of target hashes and the plurality of reference hashes, while N T and N R denote the cardinality of the plurality of target hashes and the cardinality of the plurality of reference hashes, respectively.
26. The system of claim 18 , wherein the similarity score is determined as a function of:
C/N T +C/N R
wherein C denotes the count of hashes common to both the plurality of target hashes and the plurality of reference hashes, while N T and N R denote the cardinality of the plurality of target hashes and the cardinality of the plurality of reference hashes, respectively.
27. The system of claim 18 , wherein the target object comprises a computer file.
28. The system of claim 18 , wherein the target object comprises a computer process.
29. A computer system comprising:
means for receiving a plurality of target hashes, each target hash representing a distinct code block of a target object, each distinct code block consisting of a sequence of processor instructions of the target object;
means for retrieving a plurality of reference hashes of a reference object, the reference object selected from a set of whitelisted objects according to a selected target hash of the plurality of target hashes;
means for determining a similarity score according to a count of hashes common to both the plurality of target hashes and the plurality of reference hashes; and
means for labeling the target object as non-malicious according to the similarity score.
30. A non-transitory computer-readable storage medium encoding instructions which, when executed on a processor, cause the processor to perform the steps of:
receiving a plurality of target hashes, each target hash representing a distinct code block of a target object, each distinct code block consisting of a sequence of processor instructions of the target object;
for at least a target hash of the plurality of target hashes:
retrieving a plurality of reference hashes of a reference object, the reference object selected from a set of whitelisted objects according to the target hash, and
when the plurality of target hashes is not identical to the plurality of reference hashes, determining a similarity score according to a count of hashes common to both the plurality of target hashes and the plurality of reference hashes; and
when the similarity score exceeds a predetermined threshold, labeling the target object as non-malicious.