Cluster configuration within a scalable malware detection system
A scalable, malware detection system features at least one sensor and a cluster including at least one computing node. The computing node includes an analysis coordination system and an object analysis system. The analysis coordination system, when activated as a broker computing node, (i) receives metadata from a sensor, (ii) analyzes the metadata, and (iii) places at least a portion of the metadata into a data store for subsequent use in retrieval of the suspicious object by the object analysis system from the sensor. The object analysis system is configured to (i) retrieve the portion of the metadata, which includes at least a sensor identifier, from the data store, (ii) retrieve the suspicious object from the sensor using at least part of the portion of the metadata retrieved from the data store, and (iii) analyze the suspicious object for malware.
1. A computing node comprising:
a hardware processor;
a non-transitory storage medium communicatively coupled to the hardware processor, the non-transitory storage medium comprises:
a first analysis coordinator that, when executed by the hardware processor, is configured to analyze incoming metadata received from a remotely located network device and to store at least a portion of the incoming metadata within a first data store, and
a first object analyzer that, when executed by the hardware processor, is configured to retrieve the portion of the incoming metadata from the first data store, retrieve a suspicious object associated with the incoming metadata from the network device using metadata that is part of the portion of the incoming metadata, and analyze the suspicious object for malware,
wherein the first analysis coordinator comprises monitoring logic that, when executed by the hardware processor, controls an amount of time allocated to complete malware analysis of the suspicious object, where the amount of time allocated is based, at least in part, on content within the incoming metadata and is used by the first object analyzer to determine whether to analyze the suspicious object.
2. The computing node of claim 1 , wherein the first data store operating as a queue.
3. The computing node of claim 2 , wherein the first object analyzer is further configured to determine whether the computing node features sufficient processing capacity to perform a malware analysis of the suspicious object prior to retrieval of the portion of the incoming metadata.
4. The computing node of claim 1 , wherein the first analysis coordinator comprises the monitoring logic that, when executed by the hardware processor, is configured to monitor a data store for the incoming metadata and compute a timeout value representing the amount of time that is remaining to complete the malware analysis of the suspicious object prior to storage of the portion of the incoming metadata into the first data store.
5. A computing node comprising:
a hardware processor; and
a non-transitory storage medium communicatively coupled to the hardware processor, the non-transitory storage medium comprises
a first analysis coordinator that, when executed by the hardware processor, is configured to analyze incoming metadata received from a remotely located network device and to store at least a portion of the incoming metadata within a first data store, and
a first object analyzer that, when executed by the hardware processor, is configured to retrieve the portion of the incoming metadata from the first data store, retrieve a suspicious object associated with the incoming metadata from the network device using metadata that is part of the portion of the incoming metadata, and analyze the suspicious object for malware,
wherein the first analysis coordinator comprises monitoring logic that, when executed by the hardware processor, is configured to monitor a data store for the incoming metadata and compute a timeout value representing an amount of a time period that is remaining to complete malware analysis of the suspicious object for determining whether to conduct the analysis of the suspicious object by the first object analyzer, where the timeout value is associated with the incoming metadata.
6. The computing node of claim 5 , wherein the first analysis coordinator is communicatively coupled to a sensor operating as the network device, the sensor being configured to (i) conduct a preliminary analysis of an object in the information propagating over a network by at least analyzing data associated with the object, (ii) determine whether there exists a prescribed level of likelihood that the object is associated with malware and corresponds to the suspicious object, and (iii) store the incoming metadata and the suspicious object into one or more data stores, the incoming metadata being provided to the first analysis coordinator for analysis and being subsequently used for retrieving the suspicious object from the one or more data stores by the first object analyzer.
7. The computing node of claim 6 operating as a broker computing node that establishes and maintains communications with the sensor when the first analysis coordinator is activated and being executed by the hardware processor.
8. The computing node of claim 6 , wherein the first analysis coordinator and the first object analyzer are communicatively coupled to the first data store operating as a queue, the first analysis coordinator to store at least the portion of the incoming metadata within the first data store in response to determining that the incoming metadata is not associated with an object that has been previously analyzed by a cluster including at least the first object analyzer and at least one object analyzer that is deployed as part of a different computing node than the computing node.
9. The computing node of claim 5 , wherein the first analysis coordinator comprises
pre-analysis logic that, when executed by the hardware processor, conducts an analysis of the suspicious object to determine whether a representation of the suspicious object included in the incoming metadata corresponds to any suspicious objects previously analyzed by a cluster including at least the first object analyzer and at least one object analyzer that is deployed as part of a different computing node than the computing node.
10. The computing node of claim 5 , wherein the first object analyzer further comprising:
capacity determination logic that, when executed by the hardware processor, determines whether the first object analyzer has sufficient processing capacity to handle malware analysis of the suspicious object prior to retrieval of the portion of the incoming metadata; and
access logic communicatively coupled to the capacity determination logic, the access logic, when executed by the hardware processor, retrieves the portion of the metadata from the first data store.
11. The computing node of claim 10 , wherein the first object analyzer further comprising:
content retrieval logic communicatively coupled to the access logic, the content retrieval logic, when executed by the hardware processor, commences retrieval of the suspicious object corresponding to the portion of the metadata based, at least in part, on an identifier included as part of the portion of the metadata.
12. The computing node of claim 11 , wherein the first object analyzer further comprising:
object processing logic communicatively coupled to the content retrieval logic, the object processing logic, when executed by the hardware processor, conducts the malware analysis of the suspicious object to determine whether the suspicious object is associated with malware; and
reporting logic communicatively coupled to the object processing logic, the reporting logic, when executed by the hardware processor, reports whether or not the suspicious object is associated with malware.
13. The computing node of claim 12 , wherein the object processing logic comprises
a behavior analysis logic subsystem to analyze behaviors of the suspicious object corresponding to the portion of the metadata being processed within one or more virtual machines; and
a classification logic to receive results from at least the behavior analysis logic to determine whether a likelihood of the suspicious object corresponding to the portion of the metadata being associated with malware exceeds a prescribed threshold to classify the suspicious object corresponding to the portion of the metadata as a malicious object.
14. The computing node of claim 5 , wherein the first analysis coordinator being deactivated so that the computing node operates as an analytic computing node by the first object analyzer retrieves the portion of the metadata within the first data store and conducts malware analysis on the suspicious object associated with the portion of the metadata.
15. A scalable, malware detection system comprising:
a first computing node of a plurality of computing nodes; and
a second computing node of the plurality of computing nodes communicatively coupled to the first computing node and collectively operating with the first computing node as a cluster, the cluster further includes a queue to receive metadata associated with suspicious objects queued for analysis by any of the plurality of computing nodes,
wherein the first computing node includes a first analysis coordinator being configured to (i) receive metadata associated with a first suspicious object, (ii) analyze the metadata associated with the first suspicious object to determine whether further analysis of the first suspicious object is to be conducted, (iii) responsive to determining that the first suspicious object is to be analyzed for malware, store at least a portion of the metadata in the queue that is used in an ordering of malware analyses conducted by the cluster, and (iv) monitor for incoming metadata and prior to storage of the portion of the incoming metadata in the queue, computing a timeout value representing an amount of a time period that is remaining to complete malware analysis of the first suspicious object, the timeout value being stored as a part of the metadata, and
wherein the second computing node includes a first object analyzer communicatively coupled to the first analysis coordinator, the first object analyzer being configured to (i) retrieve the portion of the metadata from the queue, (ii) retrieve the first suspicious object using at least part of the portion of the metadata, and (iii) analyze the first suspicious object for malware.
16. The system of claim 15 , wherein the first analysis coordinator of the first computing node computing the timeout value for use by the first object analyzer of the second computing node to determine whether to analyze the first suspicious object.
17. The system of claim 15 , wherein the first analysis coordinator is communicatively coupled to a sensor, the sensor being configured to (i) conduct a preliminary analysis of an object included as part of information propagating over a network by at least analyzing data associated with the object, (ii) determine whether there exists a prescribed level of likelihood that the object is associated with malware in which the object corresponds to the first suspicious object, and (iii) store incoming metadata and the first suspicious object into one or more data stores, the portion of the metadata being at least part of the incoming metadata and being subsequently used for retrieving the first suspicious object from the one or more data stores by the first object analyzer.
18. The system of claim 17 , wherein the first computing node operating as a broker computing node that establishes and maintains communications with the sensor.
19. The system of claim 15 , wherein the first computing node includes an object analyzer that is disabled.
20. The system of claim 15 , wherein the first analysis coordinator comprises
pre-analysis logic configured to conduct an analysis of the first suspicious object to determine whether a representation of the first suspicious object included in the metadata corresponds to any suspicious objects previously analyzed by the cluster.
21. The system of claim 15 , wherein the first object analyzer comprising:
capacity determination logic configured to determine whether the first object analyzer has sufficient processing capacity to handle malware analysis of the first suspicious object prior to retrieval of the portion of the metadata from the queue; and
access logic communicatively coupled to the capacity determination logic, the access logic retrieves the portion of the metadata from the queue.
22. The system of claim 21 , wherein the first object analyzer further comprising:
content retrieval logic communicatively coupled to the access logic, the content retrieval logic commences retrieval of the first suspicious object corresponding to the portion of the metadata based, at least in part, on an identifier included as part of the portion of the metadata.
23. The system of claim 22 , wherein the first object analyzer further comprising:
object processing logic communicatively coupled to the content retrieval logic, the object processing logic being configured to conduct the malware analysis of the first suspicious object to determine whether the first suspicious object is associated with malware; and
reporting logic communicatively coupled to the object processing logic, the reporting logic being configured to report whether or not the first suspicious object is associated with malware.
24. The system of claim 23 , wherein the object processing logic comprises:
a behavior analysis logic subsystem to analyze behaviors of the first suspicious object corresponding to the portion of the metadata being processed within one or more virtual machines; and
a classification logic to receive results from at least the behavior analysis logic to determine whether a likelihood of the first suspicious object corresponding to the portion of the metadata being associated with malware exceeds a prescribed threshold to classify the first suspicious object corresponding to the portion of the metadata as a malicious object.
25. The system of claim 15 , wherein the second computing node includes an analysis coordinator being deactivated so that the second computing node operates as an analytic computing node by the first object analyzer retrieving the portion of the metadata from the queue, retrieving the first suspicious object based at least on the portion of the metadata, and analyzes the first suspicious object for malware.
26. A computerized method for conducting a malware analysis of an object within a cluster, comprising:
receiving metadata associated with a first suspicious object by an analysis coordinator of a first computing node, the metadata being used by the analysis coordinator to determine whether further malware analysis is to be conducted on the first suspicious object;
responsive to determining that further malware analysis is to be conducted on the first suspicious object, computing, based on a part of the metadata, a timeout value representing an amount of a time period that is remaining to complete malware analysis of the first suspicious object and storing at least a portion of the metadata, including the timeout value, in a queue that is used, at least in part, in an ordering of malware analyses conducted by the cluster;
determining whether an object analyzer has an acceptable level of processing capacity to handle malware analysis of the first suspicious object prior to retrieval of the portion of the incoming metadata; and
retrieving the portion of the metadata from the queue by an object analyzer that is configured to (i) determine whether an object analyzer has an acceptable level of processing capacity to handle malware analysis of the first suspicious object, (ii) retrieve the portion of the metadata from the queue upon determining the object analyzer has at least the acceptable level of processing capacity to handle malware analysis of the first suspicious object, (iii) retrieve the first suspicious object using at least part of the portion of the metadata, and (iv) conduct the malware analysis on the first suspicious object.
27. The computerized method of claim 26 further comprising:
sending results of the malware analysis of the first suspicious object to a sensor for subsequent routing to a management system for dissemination to another cluster.
28. The computerized method of claim 26 , wherein the conducting of the malware analysis includes processing the first suspicious object within a virtual machine that is provisioned with one or more guest images of at least one operating system and at least one application.
29. The computerized method of claim 28 , wherein the virtual machine being provisioned with the one or more guest images to allow the virtual machine to process an object type corresponding to the first suspicious object.
30. A computing node comprising:
a hardware processor;
a non-transitory storage medium communicatively coupled to the hardware processor, the non-transitory storage medium comprises:
a first analysis coordinator that (a) supports communications with one or more remotely located network devices when the computing node is operating as a broker computing node and the first analysis coordinator is activated or (b) supports communications with another computing node that is operating as a broker computing node in lieu of any of the one or more remotely located network devices when the computing node is operating as an analytic computing node and the first analysis coordinator is deactivated, wherein the first analysis coordinator, operating as the broker computing node and executed by the hardware processor, is configured to (i) analyze incoming metadata received from the one or more remotely located network devices including determining based on the incoming metadata, a timeout value representing an amount of a time period that is remaining to complete malware analysis of a suspicious object the timeout value being part of the metadata, and (ii) store at least a portion of the incoming metadata within a first data store, and
a first object analyzer that, when executed by the hardware processor, is configured to receive a suspicious object associated with the incoming metadata and analyze the suspicious object for malware.
31. The computing node of claim 30 , wherein the first object analyzer, when activated, is further configured to determine whether the computing node features sufficient processing capacity to perform a malware analysis of the suspicious object prior to retrieval of the portion of the incoming metadata to fetch the suspicious object for analysis by the first object analyzer.
32. The computing node of claim 30 , wherein the first analysis coordinator is communicatively coupled to a sensor operating as a network device of the one or more remotely located network devices, the sensor being configured to (i) conduct a preliminary analysis of an object in the information propagating over a network by at least analyzing data associated with the object, (ii) determine whether there exists a prescribed level of likelihood that the object is associated with malware and corresponds to the suspicious object, and (iii) store the incoming metadata and the suspicious object into one or more data stores, the incoming metadata being provided to the first analysis coordinator for analysis and being subsequently used for retrieving the suspicious object from the one or more data stores by the first object analyzer.