IP Library › Granted Patent US 12,705,892
Granted Patent B2
US 12,705,892 · App. 17/971,268 · Granted Aug 11, 2026

Explainability for event alerts in video data

Inventors: Hugo Latapie (Long Beach, CA); Ozkan Kilic (Long Beach, CA); Adam James Lawrence (Pasadena, CA); Gaowen Liu (Austin, TX); Ramana Rao V. R. Kompella (Cupertino, CA)
Assignee: Cisco Technology, Inc.
G06V20/44G06V20/52G06V40/10
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,705,892
App. No.
17/971,268
Filed
Oct 21, 2022
Granted
Aug 11, 2026
Kind
B2
Examiner
HWANG, JINSU
Art Unit
2667
USPC
382/103
Abstract

In one embodiment, a device represents spatial characteristics over time of an object in video data as one or more timeseries. The device detects an event based on a rate of change of behavioral regimes associated with different portions of the one or more timeseries. The device selects contextual data for the event that comprises spatial timeseries information for different types of objects or different activities. The device provides an alert for the event to a user interface regarding the event that includes the contextual data.

Claims (33)

1 . A method comprising:

representing, by a device, spatial characteristics over time of an object in video data as one or more timeseries;

detecting, by the device, an event based on a rate of change of behavioral regimes associated with different portions of the one or more timeseries;

selecting, by the device, contextual data for the event, the contextual data comprising spatial timeseries information and data identifying one or more anomaly thresholds for different types of objects or different activities, wherein the one or more anomaly thresholds vary based on at least one of a demographic characteristic of the object or an activity being performed by the object; and

providing, by the device, an alert for the event to a user interface regarding the event that includes the contextual data, the contextual data enabling comparison of the rate of change of behavioral regimes against the one or more anomaly thresholds for the different types of objects or different activities.

2 . The method as in claim 1 , wherein the spatial characteristics comprise a detected centroid of the object.

3 . The method as in claim 1 , wherein the different types of objects comprise people having different demographics.

4 . The method as in claim 1 , wherein the different types of objects comprise different types of vehicles.

5 . The method as in claim 1 , wherein the alert is based further on the rate of change of transitions between the behavioral regimes of the object and that of one or more other objects in the video data.

6 . The method as in claim 1 , wherein the behavioral regimes are associated with the object performing different actions.

7 . The method as in claim 1 , wherein the object is a person or vehicle.

8 . The method as in claim 1 , wherein the object is a cluster of people or vehicles.

9 . The method as in claim 1 , wherein the device is an edge device in a network.

10 . An apparatus, comprising:

a network interface to communicate with a computer network;

a processor coupled to the network interface and configured to execute one or more processes; and

a memory configured to store a process that is executed by the processor, the process when executed configured to:

represent spatial characteristics over time of an object in video data as one or more timeseries;

detect an event based on a rate of change of behavioral regimes associated with different portions of the one or more timeseries;

select contextual data for the event, the contextual data comprising timeseries information and data identifying one or more anomaly thresholds for different types of objects or different activities, wherein the one or more anomaly thresholds vary based on at least one of a demographic characteristic of the object or an activity being performed by the object; and

provide an alert for the event to a user interface regarding the event that includes the contextual data, the contextual data enabling comparison of the rate of change of behavioral regimes against the one or more anomaly thresholds for the different types of objects or different activities.

11 . The apparatus as in claim 10 , wherein the spatial characteristics comprise a detected centroid of the object.

12 . The apparatus as in claim 10 , wherein the different types of objects comprise people having different demographics.

13 . The apparatus as in claim 10 , wherein the different types of objects comprise different types of vehicles.

14 . The apparatus as in claim 10 , wherein the alert is based further on the rate of change of transitions between the behavioral regimes of the object and that of one or more other objects in the video data.

15 . The apparatus as in claim 10 , wherein the behavioral regimes are associated with the object performing different actions.

16 . The apparatus as in claim 10 , wherein the object is a person or vehicle.

17 . The apparatus as in claim 10 , wherein the object is a cluster of people or vehicles.

18 . A tangible, non-transitory, computer-readable medium storing program instructions that cause a device to execute a process comprising:

representing, by the device, spatial characteristics over time of an object in video data as one or more timeseries;

detecting, by the device, an event based on a rate of change of behavioral regimes associated with different portions of the one or more timeseries;

selecting, by the device, contextual data for the event, the contextual data comprising spatial timeseries information and data identifying one or more anomaly thresholds for different types of objects or different activities, wherein the one or more anomaly thresholds vary based on at least one of a demographic characteristic of the object or an activity being performed by the object; and

providing, by the device, an alert for the event to a user interface regarding the event that includes the contextual data, the contextual data enabling comparison of the rate of change of behavioral regimes against the one or more anomaly thresholds for the different types of objects or different activities.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 21, 2022
From: LATAPIE, HUGO; KILIC, OZKAN; LAWRENCE, ADAM JAMES; LIU, GAOWEN; KOMPELLA, RAMANA RAO V. R.
To: CISCO TECHNOLOGY, INC.
Reel/Frame 061501/0357 →
Continuity (2)
Related Publication 20240135709A1 · Apr 25, 2024
Related Publication 20240233377A9 · Jul 11, 2024
References Cited (97)
US 10887197B2 · Fenoglio et al. · 2021 [cited by applicant]
US 10965516B2 · Fenoglio et al. · 2021 [cited by applicant]
US 11336506B1 · Li et al. · 2022 [cited by applicant]
US 20050180603A1 · Zoghlami et al. · 2005 [cited by applicant]
US 20080069482A1 · Komiya · 2008 [cited by applicant]
US 20100322516A1 · Xu et al. · 2010 [cited by applicant]
US 20110051992A1 · Cobb et al. · 2011 [cited by applicant]
US 20120063641A1 · Venkatesh et al. · 2012 [cited by applicant]
US 20130174116A1 · Pfeifer · 2013 [cited by applicant]
US 20150186779A1 · Deng et al. · 2015 [cited by applicant]
US 20160105617A1 · Kirkby et al. · 2016 [cited by applicant]
US 20180033024A1 · Latapie · 2018 [cited by examiner]
US 20200251091A1 · Zhao · 2020 [cited by applicant]
US 20200319715A1 · Holz · 2020 [cited by applicant]
US 20210042532A1 · Latapie et al. · 2021 [cited by applicant]
US 20210174155A1 · Smith et al. · 2021 [cited by applicant]
US 20210225409A1 · Lawlor · 2021 [cited by applicant]
US 20210258652A1 · Li et al. · 2021 [cited by applicant]
US 20210279615A1 · Latapie et al. · 2021 [cited by applicant]
US 20210312773A1 · Debnath et al. · 2021 [cited by applicant]
US 20210397849A1 · Lin et al. · 2021 [cited by applicant]
US 20230154204A1 · Kahn · 2023 [cited by examiner]
WO WO2015027289A1 · 2015 [cited by applicant]
WO WO2019168323A1 · 2019 [cited by applicant]
WO WO2021251062A1 · 2021 [cited by applicant]
Agrawal, et al., “VQA: Visual Question Answering”, Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015, 25 pages, arXiv:1505.00468v7 [cs.CL]. [cited by applicant]
Ahmed, et al., “Reflection Detection in Image Sequences”, CVPR 2011, Jun./Jul. 2011, 9 pages, IEEE, Colorado Springs, CO. [cited by applicant]
Aleksander, Igor, “Machine consciousness” In Scholarpedia. 3(2):4162, Oct. 21, 2011, 7 pages. [cited by applicant]
Anderson, et al., “Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering”, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2018, pp. 6077-6086, IEEE, Salt Lake Cit… [cited by applicant]
Baudrillard, Jean, “Simulacra and Simulation”, 1981, 159 pages, Galilee. [cited by applicant]
Baz, et al., “Context-aware hybrid classification system for fine-grained retail product recognition”, 2016 IEEE 12th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP), Jul. 2016, 5 pages, IEEE, Bord… [cited by applicant]
Bĕlohlávek, Radim, “Concept lattices and order in fuzzy logic”, Annals of Pure and Applied Logic 128 (2004) 277-298, Elsevier. [cited by applicant]
Box, G. E. P., “Science and Statistics”, In Journal of the American Statistical Association, 71(356), Dec. 1976, pp. 791-799. [cited by applicant]
Chalmers, David J., “The Conscious Mind: In Search of a Fundamental Theory”, 1996, 433 pages, Oxford University Press, New York. [cited by applicant]
Chella, et al., “A cognitive framework for imitation learning”, Robotics and Autonomous Systems 54, Mar. 2006, pp. 403-408, Elsevier. [cited by applicant]
Chella, et al., “Artificial Consciousness”, Chapter 20, in Perception-Action Cycle, 2011, pp. 637-671, Springer, New York. [cited by applicant]
Chella, et al., “Machine Consciousness: A Manifesto for Robotics”, in International Journal of Machine Consciousness, 1(1), Jun. 2009, pp. 33-51, World Scientific Publishing Company. [cited by applicant]
Cohen, Paul R., “Projections as Concepts”, Computer Science Department Faculty Publication Series (194), https://scholarworks.umass.edu/cs/_faculty/_pubs/194, 1997, 6 pages, University of Massachusetts, Amherst. [cited by applicant]
Cui, et al., “A survey on network embedding”, IEEE Transactions on Knowledge and Data Engineering, vol. 31, Issue: 5, May 1, 2019, pp. 833-852, IEEE. [cited by applicant]
D'Amour, et al., “Underspecification Presents Challenges for Credibility in Modern Machine Learning”, Underspecification in Machine Learning, online: https://arxiv.org/pdf/2011.03395.pdf, Nov. 2020, 59 pages. [cited by applicant]
De Bono, Edward, “The Mechanism of Mind”, 1967, 276 pages, Penguin Books. [cited by applicant]
Düntsch, et al., “Modal-style operators in qualitative data analysis”, 2002 IEEE International Conference on Data Mining, 2002. Proceedings, Dec. 2002, pp. 155-162, IEEE, Maebashi City, Japan. [cited by applicant]
Franco, et al., “Grocery product detection and recognition”, Expert Systems With Applications 81 (2017), pp. 163-176, Elsevier Ltd. [cited by applicant]
George, et al., “Recognizing Products: A Per-exemplar Multi-label Image Classification Approach”, ECCV 2014, Part II, LNCS 8690, 2014, pp. 440-455, Springer International Publishing Switzerland. [cited by applicant]
Goertzel, et al., “CogPrime Architecture for Embodied Artificial General Intelligence”, 2013 IEEE Symposium on Computational Intelligence for Human-like Intelligence (CIHLI), Apr. 2013, pp. 60-67, IEEE, Singapore. [cited by applicant]
Goertzel, Ben, “OpenCogPrime: A Cognitive Synergy Based Architecture for Artificial General Intelligence”, 2009 8th IEEE International Conference on Cognitive Informatics, Jun. 2009, pp. 60-68, IEEE, Hong Kong, China. [cited by applicant]
Gorban, et al., “Blessing of dimensionality: mathematical foundations of the statistical physics of data”, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 376.2118, Ja… [cited by applicant]
Grover, et al., “node2vec: Scalable Feature Learning for Networks”, KDD '16: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Aug. 2016, pp. 855-864, Association for Co… [cited by applicant]
Hamilton, et al., “Representation Learning on Graphs: Methods and Applications”, Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 2017, 23 pages, IEEE. [cited by applicant]
Hammer, et al., “A Reasoning Based Model for Anomaly Detection in the Smart City Domain”, IntelliSys 2020, AISC 1251, pp. 144-159, 2021, Springer Nature Switzerland AG. [cited by applicant]
Hobbs, Jerry R., “Granularity”, in Proceedings of the Ninth International Joint Conference on Artificial Intelligence, 1985, pp. 432-435, Morgan Kaufmann. [cited by applicant]
Horowitz, Alexandra, “Smelling themselves: Dogs investigate their own odours longer when modified in an “olfactory mirror” test”, Behavioural Processes, 2017, 41 pages. [cited by applicant]
Johnson, Mark, “The Body in The Mind”, 1987, 268 pages, The University of Chicago Press. [cited by applicant]
Jacob, et al., “A Demonstration of the Exathlon Benchmarking Platform for Explainable Anomaly Detection”, Proceedings of the VLDB Endowment (PVLDB), Oct. 2021, 5 pages, HAL Open Science. [cited by applicant]
Kiryati, et al., “A probabilistic Hough transform”, Pattern Recognition. 24(4), 1991, pp. 303-316, The Pattern Recognition Society. [cited by applicant]
Korzybski, Alfred, “Manhood of Humanity, The Science and Art of Human Engineering”, 1921, 240 pages, E. P. Dutton & Company, New York, NY. [cited by applicant]
Korzybski, Alfred, “Science and Sanity: An Introduction to Non-Aristotelian Systems and General Semantics”, 5th Edition, 1994, 910 pages, Institute of General Semantics, New York, NY. [cited by applicant]
Korzybski, Alfred, “Videos—This Is Not That”, online: https://www.thisisnotthat.com/korzybski-videos/, accessed Nov. 18, 2021, 7 pages. [cited by applicant]
Lakoff, G., “Women, Fire, and Dangerous Things”, 1984, 631 pages, University of Chicago Press. [cited by applicant]
Latapie, et al., “A Metamodel and Framework for Artificial General Intelligence from Theory to Practice”, Journal of Artificial Intelligence and Consciousness, Feb. 12, 2021, 1:30, 24 pages, World Scientific Publishing … [cited by applicant]
Li, et al., “Concept learning via granular computing: A cognitive viewpoint”, Information Sciences 298 (2015), Published Dec. 2014, pp. 447-467, Elsevier Inc. [cited by applicant]
Lieto, et al., “Conceptual Spaces for Cognitive Architectures: A Lingua Franca for Different Levels of Representation”, Biologically Inspired Cognitive Architectures 19, May 2017, 17 pages, Cognitive Robotics and Social… [cited by applicant]
Lin, et al., “Progressive Mirror Detection”, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2020, 9 pages, IEEE, Seattle, WA. [cited by applicant]
Ma, et al., “Granular computing and Dual Galois Connection”, Information Sciences, 177(23), 2007, pp. 5365-5377, Elsevier Inc. [cited by applicant]
Macaulay, Thomas, “Facebook's chief AI scientist says GPT-3 is ‘not a very good’ Q&A system”, online: https://thenextweb.com/news/facebooks-yann-lecun-says-gpt-3-is-not-very-good-as-a-qa-or-dialog-system, Oct. 28, 2020,… [cited by applicant]
Murahari, et la., “Improving Generative Visual Dialog by Answering Diverse Questions”, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on… [cited by applicant]
Park, et al., “Identifying Reflected Images from Object Detector in Indoor Environment Utilizing Depth Information”, IEEE Robotics and Automation Letters, vol. 6, No. 2, Apr. 2021, pp. 635-642, IEEE. [cited by applicant]
Patel, et al., “Video Representation and Suspicious Event Detection Using Semantic Technologies”, online: http://semantic-web-journal.net/system/files/swj2427.pdf, Semantic Web 0, Sep. 10, 2020, accessed Aug. 9, 2021, 2… [cited by applicant]
Pauli, Wolfgang, “Part I. General: (A) theory. Some relations between electrochemical behaviour and the structure of colloids”, Jan. 1935, pp. 11-27, Transactions of the Faraday Society, vol. 1. [cited by applicant]
Scarselli, et al., “The Graph Neural Network Model”, IEEE Transactions on Neural Networks (vol. 20, Issue: 1, Jan. 2009), pp. 61-80, IEEE. [cited by applicant]
Speer, et al., “ConceptNet 5.5: An Open Multilingual Graph of General Knowledge”, online: https://arxiv.org/pdf/1612.03975.pdf, 2017, 9 pagers, Association for the Advancement of Artificial Intelligence. [cited by applicant]
Swanson, Bret, “The Exponential Internet”, online: https://www.uschamberfoundation.org/bhq/exponential-internet, accessed Nov. 19, 2021, 8 pages, The U.S. Chamber of Commerce Foundation. [cited by applicant]
Tan, et al., “EfficientDet: Scalable and Efficient Object Detection”, online: https://arxiv.org/pdf/1911.09070.pdf, Jul. 2020, 10 pages. [cited by applicant]
Taylor, J. G., “CODAM: A neural network model of consciousness”, Neural Networks 20 (2007), pp. 983-992, Elsevier Ltd. [cited by applicant]
Thórisson, et al., “Cumulative Learning”, Artificial General Intelligence—12th International Conference, AGI 2019, Proceedings, pp. 198-208, Springer. [cited by applicant]
Thórisson, Kristinn R., “A New Constructivist AI: From Manual Methods to Self-Constructive Systems”, Chapter 9, Apr. 2012, pp. 147-174, Atlantis Press Book. [cited by applicant]
Thorisson, Kristinn R., “Integrated AI Systems”, Minds & Machines 17, Mar. 2007, pp. 11-25. [cited by applicant]
Tonioni, et al., “Product recognition in store shelves as a sub-graph isomorphism problem”, online: https://arxiv.org/abs/1707.08378, Sep. 2017, 14 pages. [cited by applicant]
Tripathy, et al., “Explaining Anomalies in Industrial Multivariate Time-Series Data with the Help of eXplainable AI”, 2022 IEEE International Conference on Big Data and Smart Computing (BigComp), Jan. 2022, 8 pages, IEE… [cited by applicant]
Unger, et al., “The Singular Universe and the Reality of Time: A Proposal in Natural Philosophy”, 2015, 558 pages, Cambridge University Press. [cited by applicant]
Unger, R. M. 2014. “Roberto Unger: Free Classical Social Theory from Illusions of False Necessity”, Online Lecture. 45 pages Retrieved on Nov. 22, 2021 from https://www.youtube.com/watch?v=yYOOwNRFTcY. [cited by applicant]
Wang, et al., “Concept Analysis via Rough Set and AFS Algebra”, Information Sciences 178 (2008), pp. 4125-4137, Elsevier Inc. [cited by applicant]
Wang, Pei, “Experience-grounded semantics: a theory for intelligent systems”, Aug. 2004, 33 pages, Elsevier Science. [cited by applicant]
Wang, Pei, “Insufficient Knowledge and Resources—A Biological Constraint and Its Functional Implications”, Biologically Inspired Cognitive Architectures II: Papers from the AAAI Fall Symposium (FS-09-01), 2009, pp. 188-… [cited by applicant]
Wang, Pei, “Non-axiomatic logic (nal) specification”, online: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.167.2069&rep=rep1&type=pdf, Oct. 2009, 88 pages. [cited by applicant]
Wang, Pei, “On Defining Artificial Intelligence”, Journal of Artificial General Intelligence 10(2) 2019, pp. 1-37, Sciendo. [cited by applicant]
Wang, et al. “Self in NARS, an AGI System”, vol. 5, Article 20, Mar. 2018, 15 pages, Frontiers in Robotics and AI. [cited by applicant]
Wang, et al., “SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems”, online: https://arxiv.org/pdf/1905.00537.pdf, 2019, 29 pages, 33rd Conference on Neural Information Processing Systems … [cited by applicant]
Wikipedia, “Wheat and chessboard problem”, online: https://en.wikipedia.org/wiki/Wheat_and_chessboard_problem, Oct. 2021, 5 pages, Wikimedia Foundation, Inc. [cited by applicant]
Wille, Rudolf, “Restructuring Lattice Theory: An Approach Based on Hierarchies of Concepts”, I. Rival (Ed.), Ordered Sets, 1982, pp. 314-339. [cited by applicant]
Xue, et al., “Real-Time Anomaly Detection and Feature Analysis Based on Time Series for Surveillance Video”, IEEE 5th International Conference on Universal Village . UV2020 . Session 3ABD-7, Oct. 2020, 7 pages, IEEE, Bo… [cited by applicant]
Yao, et al., “ A Granular Computing Paradigm for Concept Learning”, Emerging Paradigms in Machine Learning, Springer, London, pp. 307-326, 2012. [cited by applicant]
Yao, Y. Y., “Information Granulation and Rough Set Approximation”, International Journal of Intelligent Systems, vol. 16, No. 1, 87-104, 2001. [cited by applicant]
Yao, Y. Y., “Integrative levels of granularity”, Human-Centric Information Processing Through Granular Modelling, 2009, 20 pages, Studies in Computational Intelligence, vol. 182. Springer, Berlin, Heidelberg. [cited by applicant]
Ying, et al., “Graph convolutional neural networks for web-scale recommender systems”, online: https://arxiv.org/pdf/1806.01973.pdf, In KDD '18: The 24th ACM SIGKDD International Conference on Knowledge Discovery & Data… [cited by applicant]
Zhou, et al., “Graph neural networks: A review of methods and applications”, Al Open, 2020, pp. 57-81, Elsevier B.V. [cited by applicant]
Zhu, et al., “Describing Unseen Videos via Multi-modal Cooperative Dialog Agents” Computer Vision—ECCV 2020, 17 pages, Lecture Notes in Computer Science, vol. 12368. Springer. [cited by applicant]