Updating ground truth data in a security management platform
Updating ground truth data in a security management platform is disclosed. One example is a system including at least one processor and a memory storing instructions executable by the at least one processor to receive, in a security management platform, event data relating to a plurality of events corresponding to operation of a computing arrangement in a current time interval, and computing ground truth data for the current time interval based on the received event data, and threat intelligence data from time intervals preceding the current time interval. A prediction model is applied to generate predictions for the current time interval based on the received event data. Ground truth data is re-computed for the time intervals preceding the current time interval based on a comparison of the generated predictions and the computed ground truth data.
1. A system comprising:
a hardware processor; and
a non-transitory storage medium storing instructions executable on the hardware processor to:
receive, in a security management platform, event data relating to a plurality of events corresponding to an operation of a computing arrangement in a current time interval;
compute ground truth data for the current time interval, wherein the ground truth data is based on the received event data, and threat intelligence data from a prior time interval preceding the current time interval;
apply a first prediction model to generate a prediction for the current time interval based on the received event data;
determine a time to detection for the first prediction model, wherein the time to detection includes a time elapsed between an appearance of an entity in the computed ground truth data and a first identification of the entity as being anomalous by the first prediction model;
in response to a trigger policy being satisfied based on a determination that the time to detection for the first prediction model is less than a time to detection for a second prediction model used for making a prediction in the prior time interval, re-compute ground truth data for the prior time interval based on a comparison of the generated prediction and the computed ground truth data, wherein the second prediction model is different from the first prediction model; and
re-train, using the re-computed ground truth data, the second prediction model.
2. The system of claim 1 , wherein the instructions are executable on the hardware processor to:
generate threat intelligence data for the current time interval, wherein the threat intelligence data for the current time interval includes observables associated with malicious activities detected by the first prediction model; and
compute the ground truth data for the current time interval further based on the threat intelligence data for the current time interval.
3. The system of claim 1 , wherein the instructions are executable on the hardware processor to update a prediction for the prior time interval using the re-trained second prediction model.
4. The system of claim 3 , wherein the instructions are executable on the hardware processor to update an alert for the prior time interval based on the updated prediction.
5. The system of claim 1 , wherein the trigger policy is satisfied further based on one or more of an amount of threat intelligence data for the current time interval exceeding a first threshold, a prediction error rate for the first prediction model exceeding a second threshold, or the prediction error rate for the first prediction model trending higher over time.
6. The system of claim 1 , wherein the instructions are executable on the hardware processor to compute validation metrics for prediction models including the second prediction model in time intervals preceding the current time interval based on the re-computed ground truth data.
7. The system of claim 6 , wherein a validation metric of the validation metrics is based on factors including weighted false positives, weighted true positives, weighted false negatives, or weighted true negatives.
8. The system of claim 6 , wherein the instructions are executable on the hardware processor to compute an aggregate of the validation metrics to derive a validation metric for the first prediction model.
9. The system of claim 8 , wherein the aggregate of the validation metrics for the prediction models in the time intervals preceding the current time interval is a weighted average of the validation metrics for the prediction models in the time intervals preceding the current time interval, wherein a weight associated with a validation metric for an earlier time interval is less than a weight associated with a validation metric for a later time interval.
10. The system of claim 6 , wherein the instructions are executable on the hardware processor to use the validation metrics for determining when to re-train a prediction model of the prediction models.
11. The system of claim 1 , wherein the time to detection for the second prediction model includes a time elapsed between an appearance of a given entity in the ground truth data for the prior time interval and a first identification of the given entity as being anomalous by the second prediction model in the prior time interval.
12. A method performed by a system comprising a hardware processor, comprising:
receiving, in a security management platform, event data relating to a plurality of events corresponding to an operation of a computing arrangement in a current time interval;
training, for the current time interval, a first prediction model based on the received event data;
computing ground truth data for the current time interval, wherein the ground truth data is based on the received event data and threat intelligence data from a prior time interval preceding the current time interval;
applying the trained first prediction model to generate a prediction for the current time interval;
determining a time to detection for the trained first prediction model, wherein the time to detection includes a time elapsed between an appearance of an entity in the computed ground truth data and a first identification of the entity as being anomalous by the trained first prediction model;
re-computing ground truth data for the prior time interval in response to a trigger policy being satisfied based on a determination that the time to detection for the trained first prediction model is less than a time to detection for a second prediction model used for making a prediction in the prior time interval, wherein the second prediction model is different from the first prediction model; and
re-training, based on the re-computed ground truth data, the second prediction model.
13. The method of claim 12 , further comprising applying the trained first prediction model to compute the ground truth data for the current time interval.
14. The method of claim 12 , further comprising computing validation metrics for prediction models including the second prediction model in time intervals preceding the current time interval based on the re-computed ground truth data.
15. The method of claim 14 , further comprising computing an aggregate of the validation metrics to derive a validation metric for the first prediction model.
16. The method of claim 12 , wherein the time to detection for the second prediction model includes a time elapsed between an appearance of a given entity in the ground truth data for the prior time interval and a first identification of the given entity as being anomalous by the second prediction model in the prior time interval.
17. A non-transitory computer readable storage medium comprising instructions that upon execution cause a system to:
receive, in a security management platform, event data relating to a plurality of events corresponding to an operation of a computing arrangement;
compute ground truth data for a current time interval, wherein the ground truth data is based on the received event data and threat intelligence data from time intervals preceding the current time interval;
apply a first prediction model to generate a prediction for the current time interval based on the received event data;
determine a time to detection for the first prediction model, wherein the time to detection includes a time elapsed between an appearance of an entity in the computed ground truth data and a first identification of the entity as being anomalous by the first prediction model;
in response to a trigger policy being satisfied based on a determination that the time to detection for the first prediction model is less than a time to detection for a second prediction model used for making a prediction in a prior time interval preceding the current time interval, re-compute ground truth data for the prior time interval preceding the current time interval based on a comparison of the generated prediction and the computed ground truth data; and
re-train, using the re-computed ground truth data, the second prediction model used for making the prediction in the prior time interval, the second prediction model being different from the first prediction model.
18. The non-transitory computer readable storage medium of claim 17 , wherein the instructions upon execution cause the system to compute a validation metric for the second prediction model.
19. The non-transitory computer readable storage medium of claim 18 , wherein the instructions upon execution cause the system to use the validation metric for determining when to re-train the second prediction model.