Generating high-quality threat intelligence from aggregated threat reports
Generating high-quality threat intelligence from aggregated threat reports is provided via developing a generative model that identifies relationships between a plurality of threat assessment scanners; pre-training a plurality of individual encoders based on a corresponding plurality of pretext tasks and the generative model; combining the individual encoders into a pre-trained encoder; fine-tuning the pre-trained encoder using threat data; and marking a candidate threat, as evaluated via the pre-trained encoder as fine-tuned, as one of benign or malicious.
1 . A method, comprising:
developing a generative model that identifies relationships between a plurality of threat assessment scanners;
pre-training a plurality of individual encoders based on a corresponding plurality of pretext tasks and the generative model;
combining the individual encoders into a pre-trained encoder;
fine-tuning the pre-trained encoder using threat data; and
marking a candidate threat, as evaluated via the pre-trained encoder as fine-tuned, as one of benign or malicious; and
in response to marking the candidate threat as malicious, quarantining a computing system identified as having downloaded or accessed the candidate threat.
2 . The method of claim 1 , wherein the plurality of pretext tasks comprise:
learning dependencies between the plurality of threat assessment scanners;
modeling a temporal dynamic of threat reports from the plurality of threat assessment scanners; and
learning a temporal consistency of a plurality of embeddings for threats evaluated by the plurality of threat assessment scanners.
3 . The method of claim 1 , wherein fine-tuning is performed as a semi-supervised learning task using a plurality embeddings from the plurality of pretext tasks and a set of ground truth reports.
4 . The method of claim 1 , wherein fine-tuning is performed as an unsupervised learning task that classifies the candidate threat into one of two clusters with other entities, wherein a first cluster of the two clusters is designated as containing benign entities and a second cluster of the two clusters is designated as containing malicious entities.
5 . The method of claim 4 , wherein the first cluster of the two clusters is designated as containing the benign entities and the second cluster of the two clusters is designated as containing the malicious entities based on external verification of a subset of entities in the two clusters as being malicious or benign.
6 . The method of claim 4 , wherein the first cluster of the two clusters is designated as containing the benign entities and the second cluster of the two clusters is designated as containing the malicious entities based on the first cluster containing more entities than the second cluster.
7 . The method of claim 1 , wherein the candidate threat includes at least one of:
a phishing uniform resource locator (URL);
a malware URL;
a malware file; and
a blacklisted internet protocol address, and further comprising:
preventing a second computing device from accessing the candidate threat.
8 . A system, comprising:
a processor; and
a memory storage device including instructions that when executed by the processor perform operations comprising:
developing a generative model that identifies relationships between a plurality of threat assessment scanners;
pre-training a plurality of individual encoders based on a corresponding plurality of pretext tasks and the generative model;
combining the individual encoders into a pre-trained encoder;
fine-tuning the pre-trained encoder using threat data; and
marking a candidate threat, as evaluated via the pre-trained encoder as fine-tuned, as one of benign or malicious; and
in response to marking the candidate threat as malicious, quarantining a computing system identified as having downloaded or accessed the candidate threat.
9 . The system of claim 8 , wherein the plurality of pretext tasks comprise:
learning dependencies between the plurality of threat assessment scanners;
modeling a temporal dynamic of threat reports from the plurality of threat assessment scanners; and
learning a temporal consistency of a plurality of embeddings for threats evaluated by the plurality of threat assessment scanners.
10 . The system of claim 8 , wherein fine-tuning is performed as a semi-supervised learning task using a plurality embeddings from the plurality of pretext tasks and a set of ground truth reports.
11 . The system of claim 8 , wherein fine-tuning is performed as an unsupervised learning task that classifies the candidate threat into one of two clusters with other entities, wherein a first cluster of the two clusters is designated as containing benign entities and a second cluster of the two clusters is designated as containing malicious entities.
12 . The system of claim 11 , wherein the first cluster of the two clusters is designated as containing the benign entities and the second cluster of the two clusters is designated as containing the malicious entities based on external verification of a subset of entities in the two clusters as being malicious or benign.
13 . The system of claim 11 , wherein the first cluster of the two clusters is designated as containing the benign entities and the second cluster of the two clusters is designated as containing the malicious entities based on the first cluster containing more entities than the second cluster.
14 . The system of claim 8 , wherein the candidate threat includes at least one of:
a phishing uniform resource locator (URL);
a malware URL;
a malware file; and
a blacklisted internet protocol address, and further comprising:
preventing a second computing device from accessing the candidate threat.
15 . A non-transitory memory including instructions that when executed by a processor perform operations, the operations comprising:
developing a generative model that identifies relationships between a plurality of threat assessment scanners;
pre-training a plurality of individual encoders based on a corresponding plurality of pretext tasks and the generative model;
combining the individual encoders into a pre-trained encoder;
fine-tuning the pre-trained encoder using threat data; and
marking a candidate threat, as evaluated via the pre-trained encoder as fine-tuned, as one of benign or malicious; and
in response to marking the candidate threat as malicious, quarantining a computing system identified as having downloaded or accessed the candidate threat.
16 . The memory of claim 15 , wherein the candidate threat includes at least one of:
a phishing uniform resource locator (URL);
a malware URL;
a malware file; and
a blacklisted internet protocol address, and the operations further comprising:
preventing a second computing device from accessing the candidate threat.
17 . The memory of claim 15 , the operations further comprising:
marking a second candidate threat, as evaluated via the pre-trained encoder as fine-tuned, as one of benign or malicious; and
in response to marking the second candidate threat as benign, releasing from quarantine a second computing system identified as having downloaded or accessed the candidate threat that was quarantined in response to accessing the second candidate threat.
18 . The method of claim 1 , further comprising:
marking a second candidate threat, as evaluated via the pre-trained encoder as fine-tuned, as one of benign or malicious; and
in response to marking the second candidate threat as benign, releasing from quarantine a second computing system identified as having downloaded or accessed the candidate threat that was quarantined in response to accessing the second candidate threat.
19 . The method of claim 1 , further comprising:
marking a second candidate threat, as evaluated via the pre-trained encoder as fine-tuned, as one of benign or malicious; and
permitting a second computing system requesting access to the second candidate threat to access the second candidate threat in response to marking the second candidate threat as benign.