Configurable circular buffer for streaming multivariate ML estimation
Systems, methods, and other embodiments associated with automatic configuration of a circular buffer for ingesting a stream and generating ML estimates in real-time are described. In one embodiment, an example method includes loading a stream of multivariate time series observations into a circular buffer at a real-time pace of input from a target asset. The circular buffer is configured with a buffer configuration that specifies buffer length and choice of arrangement as a single-buffer or dual-buffer. The method then adjusts the buffer configuration until generation of machine learning estimates of the multivariate time series observations that are in the circular buffer satisfies a threshold test for generation at the real-time pace. And, at the real time pace, the method loads additional multivariate time series observations into the circular buffer that is in the adjusted configuration and generates additional machine learning estimates of the additional multivariate time series observations.
1 . One or more non-transitory computer-readable media that include stored thereon computer-executable instructions that when executed by at least a processor of a computer system cause the computer system to:
load a stream of multivariate time series observations into a circular buffer at a real-time pace of input from a target asset, wherein the circular buffer is configured with a buffer configuration that specifies a length of the circular buffer and whether the circular buffer is arranged as a single-buffer or dual-buffer;
adjust the length of the circular buffer in a memory until generation of machine learning estimates of the multivariate time series observations that are in the circular buffer satisfies a threshold test for generation at the real-time pace; and
at the real time pace, (i) load additional multivariate time series observations into the circular buffer that is in the adjusted length and (ii) generate additional machine learning estimates of the additional multivariate time series observations.
2 . The non-transitory computer-readable media of claim 1 , wherein the computer-executable instructions to adjust the length of the circular buffer in the memory causes the computer system to iteratively test the generation of the additional machine learning estimates with incremental decreases of the length of the circular buffer arranged as the single-buffer and as the dual-buffer until the threshold test is satisfied.
3 . The non-transitory computer-readable media of claim 2 , wherein the computer-executable instructions to adjust the length of the circular buffer in the memory causes the computer system to, when the length of the circular buffer reaches a pre-determined minimum:
downsample the stream of multivariate time series observations; and
reset the length of the circular buffer to a pre-determined maximum.
4 . The non-transitory computer-readable media of claim 1 , wherein the threshold test for the generation of the machine learning estimates at the real time pace is remaining within a maximum for observations missed, wherein the computer-executable instructions further cause the computer system to:
generate the machine learning estimates of the multivariate time series observations that are in the circular buffer;
write the estimates in an output stream as they are generated;
detect a count of the multivariate time series observations that are missed based on absence of estimates that correspond to the multivariate time series observations; and
determine that the threshold test for generation at the real-time pace is satisfied based on comparing the count of the multivariate time series observations that are missed with the maximum for observations missed.
5 . The non-transitory computer-readable media of claim 1 , further comprising instructions that when executed by at least the processor cause the computer system to:
detect an anomaly between the additional multivariate time series observations and the additional machine learning estimates in real time; and
in response to detection of the anomaly, generate an electronic alert in real time.
6 . The non-transitory computer-readable media of claim 1 , wherein the additional multivariate time series observations are from the target asset.
7 . The non-transitory computer-readable media of claim 1 , wherein the additional multivariate time series observations are from an asset that is of similar construction to the target asset.
8 . A configurable circular buffer system for streaming multivariate estimation, comprising:
a circular buffer that is configured to load a stream of multivariate time series observations at a real-time pace of input from a target asset, wherein the circular buffer is configured with a buffer configuration that specifies a length of the circular buffer and whether the circular buffer is arranged as a single-buffer or dual-buffer;
an estimate generator that is configured to generate machine learning estimates of the multivariate time series observations that are in the circular buffer; and
a buffer configurator that is configured to adjust the length of the circular buffer in a memory until generation of the machine learning estimates of the multivariate time series observations that are in the circular buffer satisfies a threshold test for generation at the real-time pace.
9 . The configurable circular buffer system of claim 8 , wherein the buffer configurator is further configured to iteratively test the generation of machine learning estimates with incremental decreases of the length of the circular buffer arranged as the single-buffer and as the dual-buffer until the threshold test is satisfied.
10 . The configurable circular buffer system of claim 9 , wherein the buffer configurator is further configured to to, when the length of the circular buffer reaches a pre-determined minimum:
downsample the stream of multivariate time series observations; and
reset the length of the circular buffer to a pre-determined maximum.
11 . The configurable circular buffer system of claim 8 , wherein the buffer configurator is further configured to determine that the threshold test is satisfied based on a count of observations that are missed by the estimate generator.
12 . The configurable circular buffer system of claim 8 , further comprising an anomaly detector that is configured to:
detect an anomaly between additional time series observations and corresponding additional machine learning estimates in real time; and
in response to detection of the anomaly, generate an electronic alert in real time.
13 . The configurable circular buffer system of claim 8 , wherein the estimate generator further comprises a multivariate state estimation technique model that is configured to generate the machine learning estimates to be consistent with expected values for multivariate observations of the target asset.
14 . A computer-implemented method, comprising:
configuring a circular buffer with a buffer configuration that specifies a length of the circular buffer and whether the circular buffer is arranged as a single-buffer or dual-buffer;
loading a stream of multivariate time series observations into the circular buffer at a real-time pace of input from a target asset;
adjusting the length of the circular buffer in a memory until generation of machine learning estimates of the multivariate time series observations that are in the circular buffer satisfies a threshold test for generation at the real-time pace; and
at the real-time pace,
(i) loading additional multivariate time series observations into the circular buffer that has the adjusted length and
(ii) generating additional machine learning estimates of the additional multivariate time series observations.
15 . The computer-implemented method of claim 14 , wherein adjusting the length of the circular buffer in a memory further comprises iteratively testing the generation of machine learning estimates with incremental decreases of the length of the circular buffer arranged as the single-buffer and as the dual-buffer until the threshold test is satisfied.
16 . The computer-implemented method of claim 15 , wherein adjusting the length of the circular buffer in a memory further comprises:
detecting that the length of the circular buffer has reached a pre-determined minimum;
setting the stream of multivariate time series observations to be downsampled; and
resetting the length of the circular buffer to a pre-determined maximum.
17 . The computer-implemented method of claim 14 , wherein the threshold test for generation at the real time pace is remaining within a maximum for observations missed, the method further comprising:
generating the machine learning estimates of the multivariate time series observations that are in the circular buffer;
writing the estimates in an output stream as they are generated;
detecting a count of the multivariate time series observations that are missed based on absence of estimates that correspond to the multivariate time series observations; and
determining that the threshold test for generation at the real-time pace is satisfied based on comparing the count of the multivariate time series observations that are missed with the maximum for observations missed.
18 . The computer-implemented method of claim 14 , further comprising
detecting an anomaly between the additional multivariate time series observations and the additional machine learning estimates in real time; and
in response to detection of the anomaly, generating an electronic alert in real time.
19 . The computer-implemented method of claim 14 , further comprising:
training a multivariate state estimation technique model to produce estimates that are consistent with what multivariate observations of the target asset are expected to be; and
generating the machine learning estimates of the multivariate time series observations that are in the circular buffer using the trained multivariate state estimation technique model.
20 . The computer-implemented method of claim 14 , further comprising, while loading the additional multivariate time series observations and generating the additional machine learning estimates:
load the additional multivariate time series observations into an alternative configuration of the circular buffer at the real time pace;
adjust the alternative configuration until generation of additional machine learning estimates from the additional multivariate time series observations satisfies the threshold test;
compare the alternative configuration to the adjusted buffer configuration having the adjusted length; and
in response to a difference between the adjusted buffer configuration and the alternative configuration, update the adjusted buffer configuration to the alternative configuration.