Parallel processing for malware detection
Client devices detect malware based on a ruleset received from a security server. To evaluate a current ruleset, an administrative client device initiates a ruleset evaluation of the malware detection ruleset. A security server partitions stored malware samples into a group of evaluation lists based on an evaluation policy. The security server then creates scanning nodes on an evaluation server according to the evaluation policy. The scanning nodes scan the malware samples of the evaluation lists using the rulesets and associate each malware sample with a rule of the ruleset based on the detections, if any. The security server analyzes the associations and optimizes the ruleset and stored malware samples. The security server sends the optimized ruleset to client devices such that they more efficiently detect malware samples.
1. A method for evaluating a malware detection ruleset comprising:
receiving a request to evaluate a malware detection ruleset, the request including an evaluation policy;
dividing a plurality of malware samples into a set of evaluation lists based on the evaluation policy, each evaluation list containing at least one malware sample and the evaluation lists in aggregate comprising the plurality of malware samples;
determining a number of two or more scanning nodes to install as an array of scanning nodes on an evaluation server based on the evaluation policy;
installing the array of scanning nodes on an evaluation server, the array of scanning nodes having the determined number scanning nodes, and each scanning node comprising an application for analyzing and detecting malware using the malware detection ruleset;
providing respective evaluation lists to the two or more scanning nodes of the installed array of scanning nodes;
controlling, by a processor, the array of scanning nodes to scan malware samples in the respective evaluation lists, each of the scanning nodes applying rules of the malware detection ruleset to the malware samples the respective evaluation lists;
receiving respective results from the array of the scanning nodes each indicating which rules detect which of the malware samples in the respective evaluation lists; and
storing the results to a storage medium.
2. The method of claim 1 , further comprising:
receiving an update request from the installed scanning nodes, the update request requesting a current version of the scanning node; and
sending an update to the installed scanning nodes, the update updating the scanning node to the current version of the scanning node.
3. The method of claim 1 , wherein installing the array of scanning nodes comprises:
sending an installation request to the evaluation server, wherein the installation request includes the number of scanning nodes to be created by the evaluation server and an executable image to be installed on each of the scanning nodes.
4. The method of claim 1 , further comprising:
after evaluation of the malware samples of an evaluation list by a scanning node of the array of scanning nodes, sending a subsequent evaluation list to the scanning node for evaluation.
5. The method of claim 1 wherein the evaluation policy used to divide the plurality of malware samples into a set of evaluation lists includes at least one of an amount of time to evaluate the ruleset, a size of each evaluation list, and a number of the evaluations lists.
6. The method of claim 1 , further comprising:
detecting a redundant rule in the results, the redundant rule only detecting a malware sample of the plurality of malware samples detected by at least one other rule; and
removing the redundant rule from the malware detection ruleset.
7. The method of claim 1 , further comprising:
detecting an ineffective rule in the results, the ineffective rule failing to detect a malware sample of the plurality of malware samples; and
removing the ineffective rule from the malware detection ruleset.
8. The method of claim 1 , further comprising:
detecting an unassociated malware sample, the unassociated malware sample not detected by any of the rules in the malware detection ruleset; and
storing a flag in association with the unassociated malware sample in the sample datastore.
9. The method of claim 1 wherein installing the array of scanning nodes comprises:
determining the number of scanning nodes to install such that each of the installed scanning nodes takes approximately a same amount of time to evaluate one of the respective evaluation lists.
10. The method of claim 9 , wherein the number of scanning nodes is a minimum number of scanning nodes capable of evaluating the evaluation lists within an evaluation time period included in the evaluation policy.
11. A non-transitory computer-readable storage medium storing instructions for evaluating a malware detection ruleset in a networked computer environment, the instructions when executed by one or more processors causing the one or more processors to perform steps including:
receiving a request to evaluate a malware detection ruleset, the request including an evaluation policy;
dividing a plurality of malware samples into a set of evaluation lists based on the evaluation policy, each evaluation list containing at least one malware sample and the evaluation lists in aggregate comprising the plurality of malware samples;
determining a number of two or more scanning nodes to install as an array of scanning nodes on an evaluation server based on the evaluation policy;
installing the array of scanning nodes on an evaluation server, the array of scanning nodes having the determined number of scanning nodes, and each scanning node comprising an application for analyzing and detecting malware using the malware detection ruleset;
providing respective evaluation lists to the array of scanning nodes;
controlling, by a processor, the array of scanning nodes to scan malware samples in the respective evaluation lists, each of the scanning nodes applying rules of the malware detection ruleset to the malware samples the respective evaluation lists;
receiving respective results from the array of the scanning nodes each indicating which rules detect which of the malware samples in the respective evaluation lists; and
storing the results to a storage medium.
12. The non-transitory computer-readable storage medium of claim 11 , wherein the instructions that when the one or more processors perform steps including:
determining the number of scanning nodes to install such that each of the installed scanning nodes takes approximately a same amount of time to evaluate one of the respective evaluation lists.
13. The non-transitory computer-readable storage medium of claim 11 , wherein the number of scanning nodes is a minimum number of scanning nodes capable of evaluating the evaluation lists within an evaluation time period included in the evaluation policy.
14. The non-transitory computer-readable storage medium of claim 11 , wherein the instructions that when executed by the one or more processors further perform steps including:
detecting a redundant rule in the results, the redundant rule only detecting a malware sample of the plurality of malware samples detected by at least one other rule; and
removing the redundant rule form the malware detection ruleset.
15. The non-transitory computer-readable storage medium of claim 11 , wherein the instructions that when executed by the processor further perform steps including:
detecting an ineffective rule in the results, the ineffective rule failing to detect a malware sample of the plurality of malware samples; and
removing the ineffective rule from the malware detection ruleset.
16. The non-transitory computer-readable storage medium of claim 11 , wherein the instructions that when executed by the one or more processors further perform steps including:
detecting an unassociated malware sample, the unassociated malware sample not detected by any of the rules in the malware detection ruleset; and
storing a flag in association with the unassociated malware sample in sample datastore.
17. A system comprising one or more processors processor and one or more memories storing computer program instructions for evaluating a malware detection ruleset in a networked computer environment, the instructions when executed by one or more processors causing the one or more processors to perform steps including:
receiving a request to evaluate a malware detection ruleset, the request including an evaluation policy;
dividing a plurality of malware samples into a set of evaluation lists based on the evaluation policy, each evaluation list containing at least one malware sample and the evaluation lists in aggregate comprising the plurality of malware samples;
determining a number of two or more scanning nodes to install as an array of scanning nodes on an evaluation server based on the evaluation policy;
installing the array of scanning nodes on an evaluation server, the array of scanning nodes having the determined number of scanning nodes, and each scanning node comprising an application for analyzing and detecting malware using the malware detection ruleset;
providing respective evaluation lists to the array of scanning nodes;
controlling, by a processor, the array of scanning nodes to scan malware samples in the respective evaluation lists, each of the scanning nodes applying rules of the malware detection ruleset to the malware samples the respective evaluation lists;
receiving respective results from the array of the scanning nodes each indicating which rules detect which of the malware samples in the respective evaluation lists; and
storing the results to a storage medium.
18. The system of claim 17 , wherein the instructions that when executed by the one or more processors further perform steps including:
determining the number of scanning nodes to install such that each of the installed scanning nodes takes approximately a same amount of time to evaluate one of the respective evaluation lists.
19. The system of claim 17 , wherein the number of scanning nodes is a minimum number of scanning nodes capable of evaluating the evaluation lists within an evaluation time period included in the evaluation policy.
20. The system of claim 17 , wherein the instructions that when executed by the one or more processors further perform steps including:
detecting a redundant rule in the results, the redundant rule only detecting a malware sample of the plurality of malware samples detected by at least one other rule; and
detecting an ineffective rule in the results, the ineffective rule failing to detect a malware sample of the plurality of malware samples; and
detecting an unassociated malware sample, the unassociated malware sample not detected by any of the rules in the malware detection ruleset; and
modifying the rules of the ruleset based on the detected redundant rule, ineffective rule, and unassociated malware sample.