Similarity search and malware prioritization
Methods, system, and media for determining similar malware samples are disclosed. Two or more malware samples are received and analyzed to extract information from the two or more malware samples. The extracted information is converted to a plurality of sets of strings. A similarity between the two or more malware samples is determined based on the plurality of the sets of strings.
1. A computer-implemented method comprising:
receiving two or more malware samples;
analyzing, by the one or more computer processors, the two or more malware samples to extract information from the two or more malware samples into an analyzer output, wherein the analyzer output represents relationships between portions of the extracted information in a hierarchical multi-level format;
generating, by the one or more computer processors, at least one set of strings for each of the two or more malware samples from the analyzer output by expanding hierarchical multi-level formatted information in the output into a string format, wherein the relationships between portions of the extracted information are represented as sets of strings in the string format;
determining, by the one or more computer processors, a similarity between the two or more malware samples based on the at least one set of strings for each of the two or more malware samples; and
providing, for display to a user, an output indicating the similarity between the two or more malware samples.
2. The computer-implemented method of claim 1 , wherein the at least one set of strings includes a first set of strings generated from extracted information corresponding to a first malware sample and a second set of strings generated from extracted information corresponding to a second malware sample, and
wherein determining the similarity between the two or more malware samples comprises:
determining a similarity index associated with the first set of strings and the second set of strings;
determining a distance based on the similarity index; and
determining the similarity based on the distance.
3. The computer-implemented method of claim 1 , wherein generating the at least one set of strings for each of the two or more malware samples comprises:
generating two or more respective JSONs for the two or more malware samples using the extracted information; and
generating at least one set of strings, respectively, from each of the JSONs.
4. The computer-implemented method of claim 2 , wherein determining the similarity index comprises:
determining a union of a first data set associated with the first set of strings and a second data set associated with the second set of strings;
determining an intersection of the first data set and the second data set; and
dividing the intersection by the union, and
wherein determining the distance based on the similarity index comprises:
subtracting a result of the dividing from one.
5. The computer-implemented method of claim 1 , further comprising:
obtaining a hashed set of strings from the at least one set of strings for each of the two or more malware samples, the hashed set of strings having a data size less than respective ones of the at least one set of strings for each of the two or more malware samples.
6. The computer-implemented method of claim 1 , further comprising:
determining a priority order for the two or more malware samples based, at least in part, on the similarity between the two or more malware samples.
7. A non-transitory computer-readable storage medium encoded with a computer program, the program comprising instructions that upon execution by a computer cause the computer to perform operations comprising:
receiving two or more malware samples;
analyzing the two or more malware samples to extract information from the two or more malware samples into an analyzer output, wherein the output represents relationships between portions of the extracted information in a hierarchical multi-level format;
generating at least one set of strings for each of the two or more malware samples from the analyzer output by expanding hierarchical multi-level formatted information in the analyzer output into a string format, wherein the relationships between portions of the extracted information are represented as sets of strings in the string format;
determining a similarity between the two or more malware samples based on the at least one set of strings for each of the two or more malware samples; and
providing, for display to a user, an output indicating the similarity between the two or more malware samples.
8. The medium of claim 7 , wherein the at least one set of strings includes a first set of strings generated from extracted information corresponding to a first malware sample and a second set of strings generated from extracted information corresponding to a second malware sample, and
wherein determining the similarity between the two or more malware samples comprises:
determining a similarity index associated with the first set of strings and the second set of strings;
determining a distance based on the similarity index; and
determining the similarity based on the distance.
9. The medium of claim 7 , wherein generating the at least one set of strings for each of the two or more malware samples comprises:
generating two or more respective JSONs for the two or more malware samples using the extracted information; and
generating at least one set of strings, respectively, from each of the JSONs.
10. The medium of claim 8 , wherein determining the similarity index comprises:
determining a union of a first data set associated with the first set of strings and a second data set associated with the second set of strings;
determining an intersection of the first data set and the second data set; and
dividing the intersection by the union, and
wherein determining the distance based on the similarity index comprises:
subtracting a result of the dividing from one.
11. The medium of claim 7 , wherein the operations further comprise:
obtaining a hashed set of strings from the at least one set of strings for each of the two or more malware samples, the hashed set of strings having a data size less than respective ones of the at least one set of strings for each of the two or more malware samples.
12. The medium of claim 7 , wherein the operations further comprise:
determining a priority order for the two or more malware samples based, at least in part, on the similarity between the two or more malware samples.
13. A system comprising:
one or more processors and one or more computer storage media storing instructions that are operable, when executed by the one or more processors, to cause the one or more processors to perform operations comprising:
receiving two or more malware samples;
analyzing the two or more malware samples to extract information from the two or more malware samples into an analyzer output, wherein the output represents relationships between portions of the extracted information in a hierarchical multi-level format;
generating at least one set of strings for each of the two or more malware samples from the analyzer output by expanding hierarchical multi-level formatted information in the analyzer output into a string format, wherein the relationships between portions of the extracted information are represented as sets of strings in the string format;
determining a similarity between the two or more malware samples based on the at least one set of strings for each of the two or more malware samples; and
providing, for display to a user, an output indicating the similarity between the two or more malware samples.
14. The system of claim 13 , wherein the at least one set of strings includes a first set of strings generated from extracted information corresponding to a first malware sample and a second set of strings generated from extracted information corresponding to a second malware sample, and
wherein determining the similarity between the two or more malware samples comprises:
determining a similarity index associated with the first set of strings and the second set of strings;
determining a distance based on the similarity index; and
determining the similarity based on the distance.
15. The system of claim 14 , wherein determining the similarity index comprises:
determining a union of a first data set associated with the first set of strings and a second data set associated with the second set of strings;
determining an intersection of the first data set and the second data set; and
dividing the intersection by the union, and
wherein determining the distance based on the similarity index comprises:
subtracting a result of the dividing from one.
16. The system of claim 13 , wherein generating the at least one set of strings for each of the two or more malware samples comprises:
generating two or more respective JSONs for the two or more malware samples using the extracted information; and
generating at least one set of strings, respectively, from each of the JSONs.
17. The system of claim 13 , wherein the operations further comprise:
obtaining a hashed set of strings from the at least one set of strings for each of the two or more malware samples, the hashed set of strings having a data size less than respective ones of the at least one set of strings for each of the two or more malware samples.
18. The system of claim 13 , wherein the operations further comprise:
determining a priority order for the two or more malware samples based, at least in part, on the similarity between the two or more malware samples.