Cyberanalysis workflow acceleration
A cyber threat intelligence (CTI) gateway device may receive rules for filtering TCP/IP packet communications events that are configured to cause the CTI gateway device to identify communications corresponding to indicators, signatures, and behavioral patterns of network threats. The CTI gateway device may receive packets that compose endpoint-to-endpoint communication events and, for each event, may determine that the event corresponds to criteria specified by a filtering rule. The criteria may correspond to one or more of the network threat indicators, signatures, and behavioral patterns. The CTI gateway may create a log of the threat event and forward the threat event log to a task queue managed by a cyberanalysis workflow application. Human cyberanalysts use the cyberanalysis workflow application to service the task queue by removing the task at the front of the queue, investigating the threat event, and deciding whether the event is a reportable finding that should be reported to the proper authorities. In order to improve the efficiency of the workflow process, tasks in the queue are ordered by the likelihood, or probability, that cyberanalysts will determine the associated threat events to be reportable findings; thus, high-likelihood events are investigated first likelihoods are computed using human-designed algorithms and machine-learned algorithms that are applied to characteristics of the events. Low-likelihood events may be dropped from the work queue to further improve efficiency.
1. A method comprising:
receiving a plurality of event logs;
determining, by a computing device, a reportability likelihood for each event log based on at least one algorithm, wherein the reportability likelihood for each event log is based on at least one of: a fidelity of an event threat indicator, a type of the event threat indicator, an age of the event threat indicator, threat intelligence provider data associated with the event threat indicator, reputation data of at least one threat intelligence provider, or a risk score of the event threat indicator;
sorting an event queue of the plurality of event logs based on the reportability likelihood of each of the plurality of event logs; and
transmitting, by the computing device and to an analysis system, the plurality of event logs sorted in the event queue based on the reportability likelihood of each of the plurality of event logs.
2. The method of claim 1 , wherein the reportability likelihood is a combined reportability likelihood, and wherein the determining, by the computing device, the reportability likelihood for each event log based on the at least one algorithm comprises:
determining, by the computing device, a first reportability likelihood for each event log based on a static algorithm;
determining, by the computing device, a second reportability likelihood for each event log based on a machine-learned algorithm; and
determining, by the computing device, the combined reportability likelihood for each event log based on the first reportability likelihood and the second reportability likelihood.
3. The method of claim 2 , further comprising:
receiving, from the analysis system, report data generated based on analyzed event logs; and
updating training data for a machine learning system based on the report data generated based on the analyzed event logs.
4. The method of claim 2 , wherein the machine-learned algorithm determines, for each event log, the second reportability likelihood for the event log based on at least one of: a domain name associated with the event log, an entropy value of the domain name associated with the event log, a number of labels of the domain name associated with the event log, a string length of the domain name associated with the event log, a size of data associated with the event log, or an event occurrence time associated with the event log.
5. The method of claim 2 , wherein the machine-learned algorithm is continually updated based on correlation data derived from analyzed event logs.
6. The method of claim 2 , wherein the static algorithm is a human designed algorithm, wherein the static algorithm is set based on an operator input.
7. The method of claim 1 , further comprising:
receiving, from the analysis system, report data generated based on analyzed event logs; and
updating training data for the at least one algorithm based on the report data generated based on the analyzed event logs.
8. The method of claim 1 , further comprising:
receiving a plurality of packets;
determining, based on threat intelligence data, a plurality of potential threat communications events;
generating, based on the plurality of potential threat communications events, the plurality of event logs; and
storing the plurality of event logs to the event queue.
9. The method of claim 8 , further comprising:
receiving, from the analysis system, report data generated based on analyzed event logs; and
updating, based on the report data generated based on the analyzed event logs, packet rule dispositions for determining packets to be one of the plurality of potential threat communications events.
10. The method of claim 1 , wherein the reportability likelihood is a probability that a potential threat communication is associated with an actual threat.
11. A method comprising:
receiving, by a computing device, a plurality of event logs;
determining, by the computing device, a first reportability likelihood for each event log based on a human designed algorithm;
determining, by the computing device, a second reportability likelihood for each event log based on a machine-learned algorithm;
determining, by the computing device, a combined reportability likelihood for each event log based on the first reportability likelihood and the second reportability likelihood;
sorting the plurality of event logs based on the combined reportability likelihoods of each of the plurality of event logs; and
storing, in an event queue, the plurality of event logs sorted in the event queue based on the combined reportability likelihood of each of the plurality of event logs,
wherein the combined reportability likelihood for each event log is based on at least one of: a fidelity of an event threat indicator, a type of the event threat indicator, an age of the event threat indicator, threat intelligence provider data associated with the event threat indicator, reputation data of at least one threat intelligence provider, or a risk score of the event threat indicator.
12. The method of claim 11 , further comprising:
receiving, from an analysis system, report data generated based on analyzed event logs; and
updating training data for a machine learning system based on reportability findings of analyzed event logs.
13. The method of claim 11 , wherein the machine-learned algorithm determines, for each event log, the second reportability likelihood for the event log based on at least one of: a domain name associated with the event log, an entropy value of the domain name associated with the event log, a number of labels of the domain name associated with the event log, a string length of the domain name associated with the event log, a size of data associated with the event log, or an event occurrence time associated with the event log.
14. The method of claim 11 , wherein the machine-learned algorithm determines the second reportability likelihood based on a correlation between an event and historical reportable events.
15. One or more non-transitory computer-readable media having instructions stored thereon that, when executed by one or more computing devices, cause the one or more computing devices to:
receive a plurality of event logs;
determine a reportability likelihood for each event log based on at least one algorithm, wherein the reportability likelihood for each event log is based on at least one of: a fidelity of an event threat indicator, a type of the event threat indicator, an age of the event threat indicator, threat intelligence provider data associated with the event threat indicator, reputation data of at least one threat intelligence provider, or a risk score of the event threat indicator;
sort the plurality of event logs based on the reportability likelihood of each of the plurality of event logs; and
store, in an event queue, the plurality of event logs sorted in the event queue based on the reportability likelihood of each of the plurality of event logs.
16. The one or more non-transitory computer-readable media of claim 15 , wherein the reportability likelihood is a combined reportability likelihood, the one or more non-transitory computer-readable media having instructions stored thereon to determine the reportability likelihood for each event log based on the at least one algorithm that, when executed by one or more computing devices, cause the one or more computing devices to:
determine a first reportability likelihood for each event log based on a static algorithm;
determine a second reportability likelihood for each event log based on a machine-learned algorithm; and
determine the combined reportability likelihood for each event log based on the first reportability likelihood and the second reportability likelihood.
17. The one or more non-transitory computer-readable media of claim 16 , wherein the machine-learned algorithm determines, for each event log, the second reportability likelihood for the event log based on at least one of: a domain name associated with the event log, an entropy value of the domain name associated with the event log, a number of labels of the domain name associated with the event log, a string length of the domain name associated with the event log, a size of data associated with the event log, or an event occurrence time associated with the event log.
18. The one or more non-transitory computer-readable media of claim 16 , wherein the machine-learned algorithm determines the second reportability likelihood based on a correlation between an event and historical reportable events.
19. The one or more non-transitory computer-readable media of claim 15 , having instructions stored thereon that, when executed by the one or more computing devices, further cause the one or more computing devices to:
receive, from an analysis system, report data generated based on analyzed event logs; and
update training data for the at least one algorithm based on reportability findings of analyzed event logs.
20. The one or more non-transitory computer-readable media of claim 15 , having instructions stored thereon that, when executed by the one or more computing devices, further cause the one or more computing devices to:
receive a plurality of packets;
determine, based on threat intelligence data, a plurality of potential threat communications events;
generate, based on the plurality of potential threat communications events, the plurality of event logs; and
store the plurality of event logs to the event queue.