Temporal drift detection
The present application discloses a method, system, and computer system for determining whether to train a machine learning model. The method includes analyzing a set of data for temporal drift detection, determining that a resultant stationary series has changed from training data, and in response to determining that the resultant stationary series has changed, automatically updating the machine learning model, wherein the machine learning model is trained based at least in part on a set of training data.
1 . A system, comprising:
one or more processors configured to:
obtain a stream of time-series data that is used by a machine learning model to detect network traffic anomalies;
iteratively analyze a current set of data selected from the stream in connection with detecting a temporal drift by:
processing the set of data to convert the set of data to stationary data;
removing a subset of the stationary data deemed trivial;
fitting a seasonal component of the stationary data to time series data using a boosted embeddings model;
removing seasonal data from the stationary data to obtain a resultant stationary series;
determining that a resultant stationary series has changed from training data based at least in part on a predefined threshold associated with contextual factors including one or more of time of day, network load and device configurations, wherein the threshold for determining drift is dynamically adjusted based at least in part on characteristics of the current set of data; and
in response to determining that the resultant stationary series has changed, automatically updating the machine learning model to obtain an updated machine learning model, wherein the machine learning model is trained based at least in part on a set of training data, and updating the machine learning model includes retraining the machine learning model based at least in part on the current set of data, and the machine learning model is retrained at least near real-time with respect to the detection of the temporal drift; and
deploy the updated machine learning model at a security service that is configured to use the machine learning model to detect network traffic anomalies; and
a memory coupled to the one or more processors and configured to provide the one or more processors with instructions.
2 . The system of claim 1 , wherein the one or more processors are further configured to:
train the machine learning model based at least in part on a set of training data.
3 . The system of claim 1 , wherein the one or more processors are further configured to:
receive the machine learning model.
4 . The system of claim 1 , wherein the one or more processors are further configured to:
dynamically update the machine learning model based on the temporal drift.
5 . The system of claim 4 , wherein the machine learning model is dynamically updated based on the temporal drift after a first period of time since a previous training of the machine learning model.
6 . The system of claim 4 , wherein the dynamically updated machine learning model is deployed to an enterprise security device or service.
7 . The system of claim 4 , wherein the machine learning model is not dynamically updated before detection of the temporal drift detection.
8 . The system of claim 1 , wherein the machine learning model is used in connection with detecting malicious traffic.
9 . The system of claim 1 , wherein the machine learning model is used in connection with detecting malicious domains.
10 . The system of claim 1 , wherein:
analyzing the set of data in connection with detecting the temporal drift comprises obtaining a set of time series data.
11 . The system of claim 1 , wherein the determining that the resultant stationary series has changed comprises:
determining that an extent of a difference between the resultant stationary series and the training data is greater than a preset change threshold.
12 . The system of claim 11 , wherein the training data comprises a previously resultant stationary series.
13 . The system of claim 11 , wherein the preset change threshold is three standard deviations.
14 . The system of claim 1 , wherein the boosted embeddings model fits the seasonal component by leveraging embeddings to learn seasonality or unknown cycles from multiple categorical time features including one or more of hour of day, day of week, month of year, and holiday indicators, and wherein the boosted embeddings model employs gradient boosting of deep models such that previous embeddings are frozen in each iteration while new embeddings are added to solve a residual model.
15 . The system of claim 1 , wherein determining that the resultant stationary series has changed from training data comprises computing a Z-score of the resultant stationary series with respect to the training data and determining that the Z-score exceeds the predefined threshold, and wherein the threshold is dynamically adjusted based at least in part on one or more of:
(a) variance or distribution characteristics of the current set of data;
(b) time of day or day of week at which the current set is obtained;
(c) current network load measured during the current set, and
(d) device configuration parameters associated with sources of the current set of data.
16 . The system of claim 1 , wherein the machine learning model is not automatically updated for drift before the determination that the resultant stationary series has changed.
17 . A method, comprising:
obtaining a stream of time-series data that is used by a machine learning model to detect network traffic anomalies;
iteratively analyze a current set of data selected from the stream in connection with detecting a temporal drift by:
processing the set of data to convert the set of data to stationary data;
removing a subset of the stationary data deemed trivial;
fitting a seasonal component of the stationary data to time series data using a boosted embeddings model;
removing seasonal data from the stationary data to obtain a resultant stationary series;
determining that a resultant stationary series has changed from training data based at least in part on a predefined threshold associated with contextual factors including one or more of time of day, network load and device configurations, wherein the threshold for determining drift is dynamically adjusted based at least in part on characteristics of the current set of data; and
in response to determining that the resultant stationary series has changed, automatically updating the machine learning model to obtain an updated machine learning model, wherein the machine learning model is trained based at least in part on a set of training data, and updating the machine learning model includes retraining the machine learning model based at least in part on the current set of data, and the machine learning model is retrained at least near real-time with respect to the detection of the temporal drift; and
deploying the updated machine learning model at a security service that uses the machine learning model to detect network traffic anomalies.
18 . A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:
obtaining a stream of time-series data that is used by a machine learning model to detect network traffic anomalies;
iteratively analyze a current set of data selected from the stream in connection with detecting a temporal drift by:
processing the set of data to convert the set of data to stationary data;
removing a subset of the stationary data deemed trivial;
fitting a seasonal component of the stationary data to time series data using a boosted embeddings model;
removing seasonal data from the stationary data to obtain a resultant stationary series;
determining that a resultant stationary series has changed from training data based at least in part on a predefined threshold associated with contextual factors including one or more of time of day, network load and device configurations, wherein the threshold for determining drift is dynamically adjusted based at least in part on characteristics of the current set of data; and
in response to determining that the resultant stationary series has changed, automatically updating the machine learning model to obtain an updated machine learning model, wherein the machine learning model is trained based at least in part on a set of training data, and updating the machine learning model includes retraining the machine learning model in based at least in part on the current set of data, and the machine learning model is retrained at least near real-time with respect to the detection of the temporal drift; and
deploying the updated machine learning model at a security service that uses the machine learning model to detect network traffic anomalies.