Parallel processing for malware detection
Client devices detect malware based on a ruleset received from a security server. To evaluate a current ruleset, an administrative client device initiates a ruleset evaluation of the malware detection ruleset. A security server partitions stored malware samples into a group of evaluation lists based on an evaluation policy. The security server then creates scanning nodes on an evaluation server according to the evaluation policy. The scanning nodes scan the malware samples of the evaluation lists using the rulesets and associate each malware sample with a rule of the ruleset based on the detections, if any. The security server analyzes the associations and optimizes the ruleset and stored malware samples. The security server sends the optimized ruleset to client devices such that they more efficiently detect malware samples.
1. A method for determining associations between detection rules and malware samples, the method comprising:
receiving, at an evaluation server, a set of installation instructions including:
a partition number representing the number of partitions to create on the evaluation server,
a detection ruleset for evaluating a set of malware sample subsets, each malware sample subset containing at least one malware sample and the subsets, in aggregate, comprising a plurality of malware samples, and
for each partition of the evaluation server:
installing a protection application on the partition, the protection application to scan malware sample subsets by applying rules of the detection ruleset to the malware samples of each malware sample subset;
receiving a malware sample subset of the set of malware sample subsets;
scanning the malware samples in the malware sample subset by applying rules of the detection ruleset to the malware samples in the malware sample subset;
determining an association between a malware sample of the malware sample subset and a rule of the detection ruleset, the association indicating which rules of the detection ruleset detect which malware samples in the malware sample subset;
aggregating the associations from each partition of the evaluation server into an aggregate associations list;
storing the aggregate association lists to a storage medium.
2. The method of claim 1 , further comprising:
receiving, at the evaluation server, the protection application for installation on each partition of the evaluation server from a client device, the client device requesting the evaluation of the detection ruleset against the plurality of malware samples.
3. The method of claim 1 , further comprising:
requesting, from a client device, an updated version of the protection application for installation on each partition of the evaluation server; and
in response to the request, receiving the updated version of the protection application for installation on each partition of the evaluation server.
4. The method of claim 1 , further comprising:
for a partition of the evaluation server:
in response to scanning the malware samples in the malware sample subset, receiving a new malware sample subset of the set of malware sample subsets; and
scanning the malware samples in the new malware sample subset by applying rules of the detection ruleset to the malware samples in the new malware sample subset; and
determining an new association between a malware sample of the new malware sample subset and a rule of the detection ruleset, the association indicating which rules of the detection ruleset detect which malware samples in the malware sample subset.
5. The method of claim 1 , wherein the partition number indicates the number of partitions to install on the evaluation server such that each installed protection application takes approximately a same amount of time to evaluate one malware sample subset of the set of malware sample subsets.
6. The method of claim 1 , wherein the partition number is a minimum number of partitions of on the evaluation server capable of evaluating, with installed protection applications, the set of malware sample subsets within an evaluation time period.
7. The method of claim 1 , further comprising:
receiving, at the evaluation server, the plurality of malware samples; and
dividing the plurality of malware samples into a set of malware sample subsets according to an evaluation policy, the evaluation policy including at least one of an amount of time to evaluate a malware sample subset, a size of each malware sample subset, and a number of malware sample subsets.
8. The method of claim 1 , further comprising:
detecting a redundant rule in the aggregate association lists, the redundant rule only detecting a malware sample of the plurality of malware samples detected by at least one other rule; and
removing the redundant rule from the detection ruleset.
9. The method of claim 1 , further comprising:
detecting an ineffective rule in the aggregate association lists, the ineffective rule failing to detect a malware sample of the plurality of malware samples; and
removing the ineffective rule from the detection ruleset.
10. The method of claim 1 , further comprising:
detecting an unassociated malware sample, the unassociated malware sample not detected by any of the rules in the detection ruleset; and
storing a flag in association with the unassociated malware sample in the storage medium.
11. A non-transitory computer-readable storage medium storing instructions for determining associations between detection rules and malware samples, the instructions when executed by one or more processors causing the one or more processors to perform steps including:
receiving, at an evaluation server, a set of installation instructions including:
a partition number representing the number of partitions to create on the evaluation server,
a detection ruleset for evaluating a set of malware sample subsets, each malware sample subset containing at least one malware sample and the subsets, in aggregate, comprising a plurality of malware samples, and
for each partition of the evaluation server:
installing a protection application on the partition, the protection application to scan malware sample subsets by applying rules of the detection ruleset to the malware samples of each malware sample subset;
receiving a malware sample subset of the set of malware sample subsets;
scanning the malware samples in the malware sample subset by applying rules of the detection ruleset to the malware samples in the malware sample subset;
determining an association between a malware sample of the malware sample subset and a rule of the detection ruleset, the association indicating which rules of the detection ruleset detect which malware samples in the malware sample subset;
aggregating the associations from each partition of the evaluation server into an aggregate associations list;
storing the aggregate association lists to a storage medium.
12. The non-transitory computer-readable storage medium of claim 11 , wherein the instructions, when executed by the one or more processors, further perform steps including:
receiving, at the evaluation server, the protection application for installation on each partition of the evaluation server from a client device, the client device requesting the evaluation of the detection ruleset against the plurality of malware samples.
13. The non-transitory computer-readable storage medium of claim 11 , wherein the instructions, when executed by the one or more processors, further perform steps including:
requesting, from a client device, an updated version of the protection application for installation on each partition of the evaluation server; and
in response to the request, receiving the updated version of the protection application for installation on each partition of the evaluation server.
14. The non-transitory computer-readable storage medium of claim 11 , wherein the instructions, when executed by the one or more processors, further perform steps including:
for a partition of the evaluation server:
in response to scanning the malware samples in the malware sample subset, receiving a new malware sample subset of the set of malware sample subsets; and
scanning the malware samples in the new malware sample subset by applying rules of the detection ruleset to the malware samples in the new malware sample subset; and
determining an new association between a malware sample of the new malware sample subset and a rule of the detection ruleset, the association indicating which rules of the detection ruleset detect which malware samples in the malware sample subset.
15. The non-transitory computer-readable storage medium of claim 11 , wherein the partition number indicates the number of partitions to install on the evaluation server such that each installed protection application takes approximately a same amount of time to evaluate one malware sample subset of the set of malware sample subsets.
16. The non-transitory computer-readable storage medium of claim 11 , wherein the partition number is a minimum number of partitions of on the evaluation server capable of evaluating, with installed protection applications, the set of malware sample subsets within an evaluation time period.
17. The non-transitory computer-readable storage medium of claim 11 , wherein the instructions, when executed by the one or more processors, further perform steps including:
receiving, at the evaluation server, the plurality of malware samples; and
dividing the plurality of malware samples into a set of malware sample subsets according to an evaluation policy, the evaluation policy including at least one of an amount of time to evaluate a malware sample subset, a size of each malware sample subset, and a number of malware sample subsets.
18. The non-transitory computer-readable storage medium of claim 11 , wherein the instructions, when executed by the one or more processors, further perform steps including:
detecting a redundant rule in the aggregate association lists, the redundant rule only detecting a malware sample of the plurality of malware samples detected by at least one other rule; and
removing the redundant rule from the detection ruleset.
19. The non-transitory computer-readable storage medium of claim 11 , wherein the instructions, when executed by the one or more processors, further perform steps including:
detecting an ineffective rule in the aggregate association lists, the ineffective rule failing to detect a malware sample of the plurality of malware samples; and
removing the ineffective rule from the detection ruleset.
20. The non-transitory computer-readable storage medium of claim 11 , wherein the instructions, when executed by the one or more processors, further perform steps including:
detecting an unassociated malware sample, the unassociated malware sample not detected by any of the rules in the detection ruleset; and
storing a flag in association with the unassociated malware sample in the storage medium, detecting a redundant rule in the results, the redundant rule only detecting a malware sample of the plurality of malware samples detected by at least one other rule; and
detecting an ineffective rule in the results, the ineffective rule failing to detect a malware sample of the plurality of malware samples; and
detecting an unassociated malware sample, the unassociated malware sample not detected by any of the rules in the malware detection ruleset; and
modifying the rules of the ruleset based on the detected redundant rule, ineffective rule, and unassociated malware sample.