Generating synthetic datapoints from observed datapoints for training machine learning models
An anomaly detection system is disclosed capable of reporting anomalous processes or hosts in a computer network using machine learning models trained using unsupervised training techniques. In embodiments, the system assigns observed processes to a set of process categories based on the file system path of the program executed by the process. The system extracts a feature vector for each process or host from the observation records and applies the machine learning models to the feature vectors to determine an outlier metric each process or host. The processes or hosts with the highest outlier metrics are reported as detected anomalies to be further examined by security analysts. In embodiments, the machine learnings models may be periodically retrained based on new observation records using unsupervised machine learning techniques. Accordingly, the system allows the models to learn from newly observed data without requiring the new data to be manually labeled by humans.
1. A system comprising:
one or more computing devices that implement a synthetic data generation system, configured to:
obtain a plurality of observed datapoints in a feature space encoding metadata of hosts;
select an observed datapoint from the plurality of observed datapoints;
select a direction of the synthetic datapoint relative to the observed datapoint in the feature space;
generate a plurality of synthetic datapoints in the direction with increasing distances;
stop the generation of the synthetic datapoints in response to a determination that a probability of observing a last one of the synthetic datapoints is less than a specified threshold; and
add the synthetic datapoints to a dataset, wherein the dataset is used to train or test one or more machine learning models used to analyze the metadata.
2. The system of claim 1 , wherein
the synthetic data generation system is configured to generate synthetic datapoints for the observed datapoint in multiple directions selected at random.
3. The system of claim 1 , wherein
the datapoint is a feature vector, and
the direction is selected based on a linear combination of multiple features in the feature vector.
4. The system of claim 1 , wherein
the synthetic data generation system is configured to generate synthetic datapoints for multiple ones of the observed datapoints selected at random.
5. The system of claim 1 , wherein
the observed datapoint is selected based on one or more of:
a distance of the observed datapoint from a center of the observed datapoints,
a category of the observed datapoint, and
one or more feature values of the observed datapoint.
6. The system of claim 1 , wherein
the synthetic data generation system is configured to:
compute a density function for the observed datapoints in the feature space; and
use the density function to determine probabilities of individual ones of the synthetic datapoints generated in the direction.
7. The system of claim 4 , wherein the synthetic data generation system is configured to:
compute a second density function for a plurality of datapoints generated at random; and
determine a distance range for generating the synthetic datapoints, wherein
the distance range is bounded by a first distance calculated using
the density function and a second distance calculated using
the second density function, and
the synthetic datapoints are generated within the distance range.
8. The system of claim 5 , wherein
the synthetic data generation system is configured to generate the synthetic datapoints within distance range with uniformly decreasing probabilities as determined according to the density function.
9. The system of claim 1 , wherein
the synthetic data generation system is configured to discard one or more of the synthetic datapoints based on the density function after generation of the synthetic datapoints.
10. The system of claim 1 , wherein
the synthetic datapoints are generated according to configuration information specified via a configuration of the synthetic data generation system, and
the configuration information specifies a probability range for generating the synthetic datapoints.
11. The system of claim 1 , wherein
the configuration information specifies a subset of features to use to select the direction to generate the synthetic datapoints.
12. The system of claim 1 , wherein
the observed datapoints are collected from hosts in a computer network and used to monitor the network for signals of cyberattacks.
13. The system of claim 1 , wherein
the observed datapoints are sampled from datapoints collected the computer network within a configured time window.
14. The system of claim 1 , wherein
the synthetic data generation system is configured to generate periodic synthetic datapoints periodically using different sets of observed datapoints, and
the periodic synthetic datapoints are used to update the one or more machine learning models.
15. The system of claim 1 , wherein
the synthetic datapoints are combined with the observed datapoints in the dataset,
individual datapoints in the dataset are labeled to indicate whether the individual datapoint is observed or synthetic, and
the one or more machine learning models is trained to distinguish between the observed datapoints and the synthetic datapoints.
16. A method comprising:
performing, by one or more computing devices that implement a synthetic data generation system:
obtaining a plurality of observed datapoints in a feature space encoding metadata of hosts;
selecting an observed datapoint from the plurality of observed datapoints;
selecting a direction of the synthetic datapoint relative to the observed datapoint in the feature space;
generating a plurality of synthetic datapoints in the direction with increasing distances;
stopping the generation of the synthetic datapoints in response to a determination that a probability of observing a last one of the synthetic datapoints is less than a specified threshold; and
adding the synthetic datapoints to a dataset, wherein the dataset is used to train or test one or more machine learning models used to analyze the metadata.
17. The method of claim 16 , further comprising:
the synthetic data generation system generating synthetic datapoints for the observed datapoint in multiple directions selected at random.
18. The method of claim 16 , wherein
the datapoint is a feature vector, and
the direction is selected based on a linear combination of multiple features in the feature vector.
19. The method of claim 16 , further comprising:
the synthetic data generation system generating synthetic datapoints for multiple ones of the observed datapoints selected at random.
20. The method of claim 16 , wherein
the observed datapoint is selected based on one or more of:
a distance of the observed datapoint from a center of the observed datapoints,
a category of the observed datapoint, and
one or more feature values of the observed datapoint.