IP Library › Granted Patent US 12,487,872
Granted Patent B2
US 12,487,872 · App. 18/338,222 · Granted Dec 2, 2025

Systems and methods for anomaly detection in multi-modal data streams

Inventors: Daniel Ratner (San Francisco, CA); Eric Felix Darve (Foster City, CA); Ryan Humble (San Francisco, CA)
Assignee: The Board of Trustees of the Leland Stanford Junior University
G06F11/0751G06F11/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,487,872
App. No.
18/338,222
Granted
Dec 2, 2025
Kind
B2
Abstract

Systems and method for detecting anomalies in accordance with embodiments of the invention are illustrated. One embodiment includes receiving a first data stream collected from a first sensor, identifying a first set of anomalies in the first data stream using a first model, receiving a second data stream collected from a second sensor, identifying a second set of anomalies in the second data stream using a second model, determining a set of joint anomalies using the first set of anomalies, second set of anomalies, and a threshold, wherein a threshold is some time period, and updating how anomalies are identified in the first and second set of models using the set of joint anomalies.

Claims (38)

1 . A method to train models to detect anomalies in multi-modal data streams, comprising:

receiving a first data stream collected from a first sensor;

identifying a first set of anomalies in the first data stream using a first model;

receiving a second data stream collected from a second sensor;

identifying a second set of anomalies in the second data stream using a second model;

determining a set of joint anomalies using the first set of anomalies, second set of anomalies, and a threshold, wherein a threshold is a confidence level; and

updating the first and second models using the set of joint anomalies.

2 . The method of claim 1 , wherein updating the first and second models further comprises updating a set of parameters of each of the first and second models.

3 . The method of claim 2 , further comprising updating the parameters of the first and second models to identify true positive anomalies at a rate higher than before the parameters are updated.

4 . The method of claim 1 , further comprising updating the threshold to identify true positive joint anomalies at a rate higher than before the threshold is updated.

5 . The method of claim 1 , wherein the first and second models are parameterized functions with modifiable parameters that balance the confidence level in the determined set of anomalies and a number of anomalous events predicted.

6 . The method of claim 5 , wherein the parameterized functions maximize a covariance of outputs of each of the parameterized functions.

7 . The method of claim 5 , wherein the parameterized functions maximize an unsupervised metric.

8 . The method of claim 5 , wherein the confidence level in the determined set of anomalies and the number of anomalous events are imputed from other parameters and are not directly exposed.

9 . The method of claim 1 , wherein receiving a data stream from a sensor comprises receiving data from a program that collects logs from a device.

10 . The method of claim 1 , further comprising generating output.

11 . The method of claim 10 , further comprising:

displaying the output via a graphical interface; and

sending the output over a network.

12 . The method of claim 10 , further comprising performing an event when the confidence level that a joint anomaly is detected exceeds the threshold.

13 . The method of claim 12 , wherein the event is an automatic shutoff.

14 . The method of claim 1 , further comprising:

receiving a third data stream collected from the first sensor; and

identifying a third set of anomalies in the third data stream using the updated first model.

15 . The method of claim 1 , wherein identifying the first set of anomalies is based on a set of one or more continuous functions.

16 . The method of claim 1 , wherein at least one of the first and second models is a neural network.

17 . The method of claim 1 , wherein the anomalies are represented by a group consisting of viruses, intrusions, and persistent threats.

18 . The method of claim 1 , wherein:

the first and second data streams are represented by different medical diagnostics; and

the anomalies are represented by disease diagnoses.

19 . A method of detecting manufacturing anomalies on an assembly line, comprising:

receiving a first data stream from a first sensor, wherein the first sensor sends data from a station on the assembly line;

identifying a first set of anomalies in the first data stream using a first set of models;

receiving a second data stream from a second sensor, wherein the second sensor sends data from another station on the assembly line;

identifying a second set of anomalies in the second data stream using a second set of models;

determining a set of joint anomalies using the first set of anomalies, second set of anomalies, and a threshold; and

updating how anomalies are identified in the first and second set of models using the set of joint anomalies.

20 . The method of claim 19 , wherein the second sensor sends data from a final assembly line product.

Assignments (2)
CONFIRMATORY LICENSE Recorded Sep 27, 2023
From: STANFORD UNIVERSITY
To: DEPARTMENT OF ENERGY
Reel/Frame 065046/0782 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 8, 2023
From: RATNER, DANIEL; DARVE, ERIC FELIX; HUMBLE, RYAN
To: THE BOARD OF TRUSTEES OF THE LELAND STANFORD JUNIOR UNIVERSITY
Reel/Frame 064851/0564 →
Continuity (2)
Provisional Application 63366604 · Jun 17, 2022
Related Publication 20230409422A1 · Dec 21, 2023
References Cited (30)
US 10193913B2 · Machlica · 2019 [cited by examiner]
US 11151713B2 · Kumbhare · 2021 [cited by examiner]
US 11169514B2 · Song · 2021 [cited by examiner]
US 11537116B2 · Heikkilä · 2022 [cited by examiner]
US 20150143913A1 · Adams · 2015 [cited by examiner]
US 20170220801A1 · Stockdale et al. · 2017 [cited by applicant]
US 20170230391A1 · Ferguson et al. · 2017 [cited by applicant]
US 20170230392A1 · Dean et al. · 2017 [cited by applicant]
US 20200351283A1 · Salunke · 2020 [cited by examiner]
US 20230353588A1 · Geethanath · 2023 [cited by examiner]
US 20240212350A1 · Das · 2024 [cited by examiner]
US 20240351209A1 · Graabæk · 2024 [cited by examiner]
Andrew et al., “Deep Canonical Correlation Analysis”, Proceedings of the 30th International Conference on Machine Learning, No. 3 in Proceedings of Machine Learning Research (PMLR), Jun. 17-19, 2013, pp. 1247-1255. [cited by applicant]
Breunig et al., “LOF: Identifying Density-Based Local Outliers”, ACM SIGMOD Record, vol. 29, No. 2, May 16, 2000, pp. 93-104, doi: 10.1145/335191.335388. [cited by applicant]
Challu et al., “Deep Generative model with Hierarchical Latent Factors for Time Series Anomaly Detection”, Proceedings of the 25th International Conference on Artificial Intelligence and Statistics, PMLR, vol. 151, 2022… [cited by applicant]
Chen et al., “Outlier Detection with Autoencoder Ensembles”, Proceedings of the 2017 SIAM International Conference on Data Mining, 2017, pp. 90-98, doi: 10.1137/1.9781611974973.11. [cited by applicant]
Liu et al., “Generative Adversarial Active Learning for Unsupervised Outlier Detection”, IEEE Transactions on Knowledge and Data Engineering, vol. 32, No. 8, Aug. 1, 2020, pp. 1517-1528, doi: 10.1109/TKDE.2019.2905606. [cited by applicant]
Liu et al., “Isolation Forest”, Eighth IEEE International Conference on Data Mining, Dec. 2008, pp. 413-422, doi: 10.1109/ICDM.2008.17. [cited by applicant]
Luo et al., “ViTac: Feature Sharing Between Vision and Tactile Sensing for Cloth Texture Recognition”, IEEE International Conference on Robotics and Automation (ICRA), May 2018, pp. 2722-2727, doi: 10.1109/ICRA.2018.846… [cited by applicant]
Parzen, “On Estimation of a Probability Density Function and Mode”, Annals of Mathematical Statistics, vol. 33, No. 3, Sep. 1962, pp. 1065-1076, doi: 10.1214/aoms/1177704472. [cited by applicant]
Reynolds, “Gaussian mixture models”, Encyclopedia of Biometrics. Springer, 2009, pp. 659-663, doi: 10.1007/978-0-387-73003-5_196. [cited by applicant]
Ruff et al., “Deep One-Class Classification”, Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden, PMLR, vol. 80, Jul. 10-15, 2018, pp. 4393-4402. [cited by applicant]
Schlegl et al., “f-AnoGAN: Fast unsupervised anomaly detection with generative adversarial networks”, Medical Image Analysis, vol. 54, May 2019, pp. 30-44, doi: 10.1016/j.media.2019.01.010. [cited by applicant]
Schlegl et al., “Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery”, Information Processing in Medical Imaging, vol. 10265, May 23, 2017, pp. 146-157, doi: 10.1007/978-3-319-5… [cited by applicant]
Schölkopf et al., “Estimating the Support of a High-Dimensional Distribution”, Neural Computation, vol. 13, No. 7, Jul. 2001, pp. 1443-1471, doi: 10.1162/089976601750264965. [cited by applicant]
Shen et al., “Timeseries Anomaly Detection using Temporal Hierarchical One-Class Network”, NIPS'20: Proceedings of the 34th International Conference on Neural Information Processing Systems, vol. 1092, Dec. 6, 2020, pp.… [cited by applicant]
Su et al., “Robust Anomaly Detection for Multivariate Time Series through Stochastic Recurrent Neural Network”, Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining—KDD '19, A… [cited by applicant]
Yang et al., “Regularized Cycle Consistent Generative Adversarial Network for Anomaly Detection”, 24th European Conference on Artificial Intelligence (ECAI), IOS Press, 2020, pp. 1618-1625, doi: 10.3233/FAIA200272. [cited by applicant]
Zenati et al., “Adversarially Learned Anomaly Detection”, Proceedings of the IEEE International Conference on Data Mining, Nov. 17-20, 2018, pp. 727-736, doi: 10.1109/ICDM.2018.00088. [cited by applicant]
Zhou et al., “VAE-based Deep SVDD for anomaly detection”, Neurocomputing, vol. 453, Sep. 17, 2021, pp. 131-140, doi: 10.1016/j.neucom.2021.04.089. [cited by applicant]