System and method for high speed threat intelligence management using unsupervised machine learning and prioritization algorithms
This document discloses a system and method for consolidating threat intelligence data for a computer and its related networks. Massive volumes of raw threat intelligence data are collected from a plurality of sources and are partitioned into a common format for cluster analysis whereby the clustering of the data is done using unsupervised machine learning algorithms. The resulting organized threat intelligence data subsequently undergoes a weighted asset based threat severity level correlation process. All the intermediary network vulnerabilities of a particular computer network are utilized as the critical consolidation parameters of this process. The final processed intelligence data gathered through this high speed automated process is then formatted into predefined formats prior to transmission to third parties.
1. A method of consolidating threat intelligence data for a computer network, the method to be performed by a computer system comprising: collecting threat intelligence data from a plurality of sources and normalizing the collected threat intelligence data into a uniform data format; grouping normalized threat intelligence data into clusters using unsupervised machine learning algorithms, wherein each cluster comprises a group of data that represents an attribute of the threat intelligence data; categorizing clusters that are severe to the computer network; comparing the clusters categorized as severe with a security posture of the computer network to determine clusters that are of interest to the computer system; and formatting the clusters determined to be of interest to the computer system to a predefined format of the computer network; and
wherein the categorizing the clusters that are severe to the computer network comprises: retrieving a list of computer assets associated with the computer network; identifying clusters that affect a computing feature of the computer assets; and classifying identified clusters that affect a computing feature of the computer asset as severe to the computer network.
2. The method according to claim 1 further comprising retrieving severity weightage values accorded to each of the computer assets associated with the computer network; summing the retrieved severity weightage values; and allocating the summed severity weightage value to the computer network.
3. The method according to claim 1 wherein the computing feature comprises an operating system or a network protocol of a computer asset.
4. The method according to claim 1 , wherein before the comparing the clusters categorized as severe with a security posture of the computer network to determine clusters of interest to the computer system, the method further comprises: generating the security posture of the computer network.
5. The method according to claim 4 wherein the generating the security posture of the computer network comprises: creating an object model representing the computer network, wherein the object model includes computer security information of computer assets contained within the computer network; and executing an analysis program operative to run vulnerability testing of each of the computer assets in the computer network using the object model, wherein the results of the vulnerability testing are used to determine the security posture of the computer network.
6. The method according to claim 5 , wherein the vulnerability testing of each of the computer assets in the computer network using the object model comprises tests pertaining to system level and topology vulnerabilities of the computer network, and node level vulnerabilities of the computer assets.
7. The method according to claim 1 wherein the grouping normalised threat intelligence data into clusters machine learning algorithms, wherein each cluster comprises a group of data that represents an attribute of the threat intelligence data further comprises: validating the clusters using threat intelligence data in each cluster.
8. The method according to claim 7 wherein the validating the clusters comprises: assigning weightage values to each record contained in the clusters, wherein a record originating from an open source is assigned a lower weightage value as compared to a weightage value assigned to a record originating from a commercial source; summing the weightage values of records contained in each cluster; and validating clusters that have summed weightage values that exceed a predefined threshold.
9. The method according to claim 1 further comprising: using the formatted clusters to update the security posture of the computer network.
10. The method according to claim 1 wherein the attribute of the threat intelligence data comprises a computer security threat or an Internet Protocol (IP) address.
11. A system for consolidating threat intelligence data for a computer network comprising: a processing unit; and a non-transitory media readable by the processing unit, the media storing instructions that when executed by the processing unit, cause the processing unit to; collect threat intelligence data from a plurality of sources and normalize the collected threat intelligence data into a uniform data format; group normalized threat intelligence data into clusters using unsupervised machine learning algorithms, wherein each cluster comprises a group of data that represents an attribute of the threat intelligence data; categorize dusters that are severe to the computer network; compare the clusters categorized as severe with a security posture of the computer network to determine clusters that are of interest to the computer system; and format the clusters determined to be of interest to the computer system to a predefined format of the computer network; and
wherein the instructions to categorize the clusters that are severe to the computer network comprises: instructions for directing the processing unit to: retrieve a list of computer assets associated with the computer network; identify dusters that affect a computing feature of the computer assets; and classify identified clusters that affect a computing feature of the computer asset as severe to the computer network.
12. The system according to claim 11 further comprising: instructions for directing the processing unit to: retrieve severity weightage values accorded to each of the computer assets associated with the computer network; sum the retrieved severity weightage values; and allocate the summed severity weightage value to the computer network.
13. The system according to claim 11 wherein the computing feature comprises an operating system or a network protocol of a computer asset.
14. The system according to claim 11 , wherein before the instructions to compare the clusters categorized as severe with a security posture of the computer network to determine clusters of interest to the computer system, the system further comprises: instructions for directing the processing unit to: generate the security posture of the computer network.
15. The system according to claim 14 wherein the instructions to generate the security posture of the computer network comprises: instructions for directing the processing unit to: create an object model representing the computer network, wherein the object model includes computer security information of computer assets contained within the computer network; and execute an analysis program operative to run vulnerability testing of each of the computer assets in the computer network using the object model, wherein the results of the vulnerability testing are used to determine the security posture of the computer network.
16. The system according to claim 15 , wherein the vulnerability testing of each of the computer assets in the computer network using the object model comprises tests pertaining to system level and topology vulnerabilities of the computer network, and node level vulnerabilities of the computer assets.
17. The system according to claim 11 wherein the instructions to group normalised threat intelligence data into clusters machine learning algorithms, wherein each cluster comprises a group of data that represents an attribute of the threat intelligence data further comprises: instructions for directing the processing unit to: validate the clusters using threat intelligence data in each cluster.
18. The system according to claim 17 wherein the instructions to validate the clusters comprises: instructions for directing the processing unit to: assign weightage values to each record contained in the clusters, wherein a record originating from an open source is assigned a lower weightage value as compared to a weightage value assigned to a record originating from a commercial source; sum the weightage values of records contained in each cluster; and validate clusters that have summed weightage values that exceed a predefined threshold.
19. The system according to claim 11 further comprising: instructions for directing the processing unit to: use the formatted clusters to update the security posture of the computer network.
20. The system according to claim 11 wherein the attribute of the threat intelligence data comprises a computer security threat or an Internet Protocol (IP) address.
21. A system for consolidating threat intelligence data for a computer network comprising: circuitry configured to collect threat intelligence data from a plurality of sources and normalize the collected threat intelligence data into a uniform data format; circuitry configured to group normalized threat intelligence data into clusters using unsupervised machine learning algorithms, wherein each cluster comprises a group of data that represents an attribute of the threat intelligence data; circuitry configured to categorize clusters that are severe to the computer network; circuitry configured to compare the clusters categorized as severe with a security posture of the computer network to determine clusters that are of interest to the computer system; and circuitry configured to format the clusters determined to be of interest to the computer system to a predefined format of the computer network; and
wherein the circuitry configured to categorize the clusters that are severe to the computer network comprises: circuitry configured to retrieve a list of computer assets associated with the computer network; circuitry configured to identify clusters that affect a computing feature of the computer assets; and circuitry configured to classify identified clusters that affect a computing feature of the computer asset as severe to the computer network.
22. The system according to claim 21 further comprising: circuitry configured to retrieve severity weightage values accorded to each of the computer assets associated with the computer network; circuitry configured to sum the retrieved severity weightage values; and circuitry configured to allocate the summed severity weightage value to the computer network.
23. The system according to claim 21 wherein the computing feature comprises an operating system or a network protocol of a computer asset.
24. The system according to claim 21 , wherein before the circuitry configured to compare the clusters categorized as severe with a security posture of the computer network to determine clusters of interest to the computer system, the system further comprises: circuitry configured to generate the security posture of the computer network.
25. The system according to claim 24 wherein the circuitry configured to generate the security posture of the computer network comprises: circuitry configured to create an object model representing the computer network, wherein the object model includes computer security information of computer assets contained within the computer network; and circuitry configured to execute an analysis program operative to run vulnerability testing of each of the computer assets in the computer network using the object model, wherein the results of the vulnerability testing are used to determine the security posture of the computer network.
26. The system according to claim 25 , wherein the vulnerability testing of each of the computer assets in the computer network using the object model comprises tests pertaining to system level and topology vulnerabilities of the computer network, and node level vulnerabilities of the computer assets.
27. The system according to claim 21 wherein the circuitry configured to group normalized threat intelligence data into clusters machine learning algorithms, wherein each cluster comprises a group of data that represents an attribute of the threat intelligence data further comprises: circuitry configured to validate the clusters using threat intelligence data in each cluster.
28. The system according to claim 27 wherein the circuitry configured to validate the clusters comprises: circuitry configured to assign weightage values to each record contained in the clusters, wherein a record originating from an open source is assigned a lower weightage value as compared to a weightage value assigned to a record originating from a commercial source; circuitry configured to sum the weightage values of records contained in each cluster; and circuitry configured to validate clusters that have summed weightage values that exceed a predefined threshold.
29. The system according to claim 21 further comprising: circuitry configured to use the formatted clusters to update the security posture of the computer network.
30. The system according to claim 21 wherein the attribute of the threat intelligence data comprises a computer security threat or an Internet Protocol (IP) address.