High-confidence malware severity classification of reference file set

1. A method of generating a reference file set having high-confidence malware severity classification, comprising:

selecting a subset of files from a group of files first observed during a recent observation period;

polling a plurality of other antivirus providers for their third-party classification of the files in the subset of files and for their third-party classification of a plurality of files from the group of files not in the subset;

determining a malware severity classification for the files in the subset by aggregating the polled classifications from the other antivirus providers for the files in the subset after a stabilization period of time; and

adding one or more files having a third-party classification from at least one of the polled other antivirus providers that changed during the stabilization period to the subset.

2. The method of generating a reference file set having high-confidence malware severity classification of claim 1 , further comprising adding one or more files to the subset of files selected to improve representation of malware types seen during recent observation period in a distribution of malware types in the subset.

3. The method of generating a reference file set having high-confidence malware severity classification of claim 2 , wherein malware types comprise at least one of malware families, malware having similar functions, and malware having similar severity.

4. The method of generating a reference file set having high-confidence malware severity classification of claim 1 , further comprising determining a malware severity classification for the one or more added files having a third-party classification that changed during the stabilization by aggregating the polled classifications from the other antivirus providers for the one or more added files.

5. The method of generating a reference file set having high-confidence malware severity classification of claim 1 , further comprising assigning the subset and malware severity classifications for the subset as immutable truth for the recent observation period for purposes of future testing.

6. The method of generating a reference file set having high-confidence malware severity classification of claim 1 , wherein the recent observation period comprises a day, three days, or a week.

7. The method of generating a reference file set having high-confidence malware severity classification of claim 1 , wherein selecting the subset of files from the group of files is done randomly.

8. The method of generating a reference file set having high-confidence malware severity classification of claim 1 , wherein the stabilization period comprises a period of two days to one week.

9. The method of generating a reference file set having high-confidence malware severity classification of claim 1 , wherein determining classification further comprises using at least one of majority voting, statistical estimation, and machine learning using third-party classification of the files from the polled third-party antivirus providers.

10. The method of generating a reference file set having high-confidence malware severity classification of claim 1 , wherein adding one or more files having a third-party classification from at least one of the polled other antivirus providers that changed during the stabilization period to the subset comprises adding all files having a third-party classification from at least one of the polled other antivirus providers that changed during the stabilization period.

11. The method of generating a reference file set having high-confidence malware severity classification of claim 1 , wherein adding one or more files having a third-party classification from at least one of the polled other antivirus providers that changed during the stabilization period to the subset increases the percentage of files in the subset that are more difficult to classify, thereby improving complexity of the subset.

12. A method of estimating the effectiveness of an anti-malware algorithm, comprising:

testing the anti-malware algorithm against a reference file set; and

evaluating the accuracy of the malware algorithm in characterizing a malware severity of each file in the reference set;

wherein the reference file set is constructed by:

selecting a subset of files from a group of files first observed during a recent observation period;

polling a plurality of other antivirus providers for their third-party classification of the files in the subset of files and for their third-party classification of a plurality of files from the group of files not in the subset;

determine a malware severity classification for the files in the subset by aggregating the polled classifications from the other antivirus providers for the files in the subset after a stabilization period of time; and

adding one or more files having a third-party classification from at least one of the polled other antivirus providers that changed during the stabilization period to the subset.

13. The method of estimating the effectiveness of an anti-malware algorithm of claim 12 , wherein the reference file set is further constructed by adding one or more files to the subset of files selected to improve representation of malware types seen during recent observation period in a distribution of malware types in the subset.

14. The method of estimating the effectiveness of an anti-malware algorithm of claim 12 , wherein the reference file set is further constructed by determining a malware severity classification for the one or more added files having a third-party classification that changed during the stabilization by aggregating the polled classifications from the other antivirus providers for the one or more added files.

15. The method of estimating the effectiveness of an anti-malware algorithm of claim 12 , wherein the recent observation period comprises a period of one day to one week, and the stabilization period comprises a period of two days to one week.

16. A method of generating a reference file set having high-confidence malware severity classification, comprising:

selecting a subset of files from a group of files first observed during a recent observation period;

polling a plurality of other antivirus providers for their third-party classification of the files in the subset of files and for their third-party classification of a plurality of files from the group of files not in the subset;

determining a malware severity classification for the files in the subset by aggregating the polled classifications from the other antivirus providers for the files in the subset after a stabilization period of time;

adding one or more first additional files to the subset having a third-party classification from at least one of the polled other antivirus providers that changed during the stabilization period;

adding one or more second additional files to the subset selected to improve representation of malware types seen during recent observation period in a distribution of malware types in the subset;

determining a malware severity classification for the one or more first and second additional files added to the subset by aggregating the polled classifications from the other antivirus providers for the added files.

17. The method of generating a reference file set having high-confidence malware severity classification of claim 16 , further comprising assigning the subset and malware severity classifications for the subset as immutable truth for the recent observation period for purposes of future testing.

18. The method of generating a reference file set having high-confidence malware severity classification of claim 16 , wherein the recent observation period comprises a period of one day to one week, and the stabilization period comprises a period of two days to one week.

19. The method of generating a reference file set having high-confidence malware severity classification of claim 16 , wherein adding one or more files having a third-party classification from at least one of the polled other antivirus providers that changed during the stabilization period to the subset comprises adding all files having a third-party classification from at least one of the polled other antivirus providers that changed during the stabilization period.

20. The method of generating a reference file set having high-confidence malware severity classification of claim 16 , wherein adding one or more first additional files having a third-party classification from at least one of the polled other antivirus providers that changed during the stabilization period to the subset increases the percentage of files in the subset that are more difficult to classify, thereby improving complexity of the subset.