Reducing malware signature redundancy
Redundancy in a malware signature list is reduced by processing a plurality of pairs of records in a known malware signature list, where each pair of records comprises a file identifier and an associated malware detection. At least one of the file identifiers and the associated malware detections are mapped to symbols representing the file identifiers and the associated malware detections, the symbols taking less memory than the file identifiers and the associated malware detections. The mapped symbols representing the file identifiers and the associated malware detections are processed to remove at least some malware detections that are not needed to provide a desired degree of representation of each file identifier in the processed known malware signature list, and a processed known malware signature list is stored.
1. A method of reducing redundancy in a malware signature list, comprising:
reading a plurality of pairs of records in the known malware signature list, each pair of records comprising a file identifier and an associated malware detection;
mapping at least one of the file identifiers and the associated malware detections to symbols representing the at least one of the file identifiers and the associated malware detections, the symbols taking less memory than the at least one of the file identifiers and the associated malware detections;
evaluating the mapped file identifiers for multiple detections of known malware represented in the file identifiers;
processing the mapped symbols representing the at least one of the file identifiers and the associated malware detections to remove at least some malware detections from the processed known malware signature list that are not needed to provide a desired degree of representation of more than one of each file identifier in the processed known malware signature list, wherein the degree of representation of a known malware in the known malware signature list is determined by the number of redundant detections of the known malware; and
storing a record of the processed known malware signature list.
2. The method of reducing redundancy in a malware signature list of claim 1 , wherein the desired degree of representation of each file identifier comprises covering each file by at least one detection.
3. The method of reducing redundancy in a malware signature list of claim 2 , wherein the desired degree of representation of each file identifier further comprises employing an algorithm to determine which of a plurality of detections covering a file should be retained.
4. The method of reducing redundancy in a malware signature list of claim 1 , wherein the desired degree of representation of each file identifier is that each file be represented by at least a configurable number of malware detections before additional malware detections covering the file are removed.
5. The method of reducing redundancy in a malware signature list of claim 1 , wherein the file identifier is a hash of the file.
6. The method of reducing redundancy in a malware signature list of claim 1 , wherein the desired degree of representation of each file identifier comprises not removing detections that cover more than a threshold number of unique files.
7. The method of reducing redundancy in a malware signature list of claim 1 , wherein the mapping at least one of the file identifiers and the associated malware detections occurs for each of the plurality of pairs of records before the processing the mapped symbols representing the at least one of the file identifiers and the associated malware detections to remove at least some malware detections.
8. The method of reducing redundancy in a malware signature list of claim 1 , wherein the malware detection comprises a name of the malware in the associated file.
9. The method of reducing redundancy in a malware signature list of claim 1 , wherein the symbols representing the at least one of the file identifiers and the associated malware detections comprise numbers.
10. The method of reducing redundancy in a malware signature list of claim 1 , wherein the mapped symbols representing the at least one of the file identifiers and the associated malware detections are stored in nonvolatile storage.
11. The method of reducing redundancy in a malware signature list of claim 10 , wherein the mapped symbols representing the at least one of the file identifiers and the associated malware detections stored in nonvolatile storage are read into memory in segments for the processing to remove at least some malware detections.
12. The method of reducing redundancy in a malware signature list of claim 1 , further comprising creating and storing a map for the at least one of the mapped file identifiers and the associated malware detections.
13. The method of reducing redundancy in a malware signature list of claim 1 , wherein mapping at least one of the file identifiers and the associated malware detections to symbols representing the at least one of the file identifiers and the associated malware detections comprises mapping the file identifiers to symbols representing the file identifiers and mapping the associated malware detections to symbols representing the associated malware detections.
14. A computerized system, comprising:
a processor;
a memory;
nonvolatile storage; and
instructions stored on the nonvolatile storage, the instructions operable when executed on the processor to cause the computerized system to:
map a plurality of pairs of file identifiers and associated malware detections from a known malware signature list to symbols representing the file identifiers and symbols representing the associated malware detections, the symbols taking less memory than the file identifiers and the associated malware detections;
evaluating the mapped file identifiers for multiple detections of known malware represented in the file identifiers; and
process the mapped plurality of pairs to remove at least some malware detections from a processed known malware signature list that are not needed to provide a desired degree of representation of more than one of each file identifier in a processed known malware signature list, wherein the degree of representation of a known malware in the known malware signature list is determined by the number of redundant detections of the known malware.
15. The computerized system of claim 14 , wherein the desired degree of representation of each file identifier is that one of a plurality of detections covering a file should be retained as selected by an algorithm.
16. The computerized system of claim 14 , wherein the desired degree of representation of each file identifier is that each file be represented by at least a configurable number of malware detections before additional malware detections covering the file are removed.
17. The computerized system of claim 14 , wherein the desired degree of representation of each file identifier comprises not removing detections that cover more than a threshold number of unique files.
18. A method of reducing redundancy in a malware signature list, comprising:
mapping a plurality of pairs of file identifiers and associated malware detections from a known malware signature list to symbols representing the file identifiers and symbols representing the associated malware detections, the symbols taking less memory than the file identifiers and the associated malware detections;
evaluating the mapped file identifiers for multiple detections of known malware represented in the file identifiers; and
processing the mapped plurality of pairs to remove at least some malware detections from a processed known malware signature list that are not needed to provide a desired degree of representation of each file identifier in the processed known malware signature list, wherein the degree of representation of a known malware in the known malware signature list is determined by the number of redundant detections of the known malware.