IP Library › Granted Patent US 12,731,267
Granted Patent B2
US 12,731,267 · App. 17/971,259 · Granted Sep 8, 2026

Spatial regime change detection for video analytics

Inventors: Hugo Latapie (Long Beach, CA); Ozkan Kilic (Long Beach, CA); Adam James Lawrence (Pasadena, CA); Gaowen Liu (Austin, TX); Ramana Rao V. R. Kompella (Cupertino, CA)
Assignee: Cisco Technology, Inc.
G06T7/20G06V20/41G06T2207/10016G06T2207/30196G06T2207/30232G06V2201/08
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,731,267
App. No.
17/971,259
Granted
Sep 8, 2026
Kind
B2
Abstract

In one embodiment, a device tracks an object in video data captured by one or more cameras. The device represents spatial characteristics of the object over time as one or more timeseries. The device associates different portions of the one or more timeseries with behavioral regimes of the object. The device provides a portion of the video data for display as part of an alert that is based on a rate of change of transitions between the behavioral regimes of the object.

Claims (49)

1 . A method comprising:

tracking, by a device, an object in video data captured by one or more cameras;

representing, by the device, spatial characteristics of the object over time as one or more timeseries;

associating, by the device, different portions of the one or more timeseries with behavioral regimes of the object, wherein the behavioral regimes comprise different patterns of the one or more timeseries associated with the object performing different actions;

detecting, by the device, transitions between the behavioral regimes by identifying changes in a behavioral-regime association for successive portions of the one or more timeseries;

based on the changes, computing a derivative of a transition count that quantifies a rate of change in a number of transitions detected within a time window between the behavioral regimes of the object;

comparing the derivative to one or more threshold values;

based on the derivative exceeding at least one of the one or more threshold values, generating a regime change alert; and

providing, by the device, the regime change alert and a portion of the video data for display as part of the regime change alert.

2 . The method as in claim 1 , wherein the spatial characteristics comprise a detected centroid of the object.

3 . The method as in claim 1 , wherein the portion of the video data is provided for display with an associated timestamp.

4 . The method as in claim 1 , wherein the portion of the video data depicts at least one transition between the behavioral regimes.

5 . The method as in claim 1 , wherein the alert is based further on the rate of change of transitions between the behavioral regimes of the object relative to that of one or more other objects in the video data.

6 . The method as in claim 5 , further comprising:

tracking the one or more other objects in the video data.

7 . The method as in claim 1 , wherein the object is a person or vehicle.

8 . The method as in claim 1 , wherein the object is a cluster of people or vehicles.

9 . The method as in claim 1 , wherein the device is an edge device in a network.

10 . An apparatus, comprising:

a network interface to communicate with a computer network;

a processor coupled to the network interface and configured to execute one or more processes; and

a memory configured to store a process that is executed by the processor, the process when executed configured to:

track an object in video data captured by one or more cameras;

represent spatial characteristics of the object over time as one or more timeseries;

associate different portions of the one or more timeseries with behavioral regimes of the object, wherein the behavioral regimes comprise different patterns of the one or more timeseries associated with the object performing different actions;

detect transitions between the behavioral regimes by identifying changes in a behavioral-regime association for successive portions of the one or more timeseries;

based on the changes, compute a derivative of a transition count that quantifies a rate of change in a number of transitions detected within a time window between the behavioral regimes of the object;

compare the derivative to one or more threshold values;

based on the derivative exceeding at least one of the one or more threshold values, generating a regime change alert; and

provide, by the device, the regime change alert and a portion of the video data for display as part of the regime change alert.

11 . The apparatus as in claim 10 , wherein the spatial characteristics comprise a detected centroid of the object.

12 . The apparatus as in claim 10 , wherein the portion of the video data is provided for display with an associated timestamp.

13 . The apparatus as in claim 10 , wherein the portion of the video data depicts at least one transition between the behavioral regimes.

14 . The apparatus as in claim 10 , wherein the alert is based further on the rate of change of transitions between the behavioral regimes of the object relative to that of one or more other objects in the video data.

15 . The apparatus as in claim 14 , wherein the process when executed is further configured to:

track the one or more other objects in the video data.

16 . The apparatus as in claim 10 , wherein the object is a person or vehicle.

17 . The apparatus as in claim 10 , wherein the object is a cluster of people or vehicles.

18 . A tangible, non-transitory, computer-readable medium storing program instructions that cause a device to execute a process comprising:

tracking, by the device, an object in video data captured by one or more cameras;

representing, by the device, spatial characteristics of the object over time as one or more timeseries;

associating, by the device, different portions of the one or more timeseries with behavioral regimes of the object, wherein the behavioral regimes comprise different patterns of the one or more timeseries associated with the object performing different actions;

detecting, by the device, transitions between the behavioral regimes by identifying changes in a behavioral-regime association for successive portions of the one or more timeseries;

based on the changes, computing a derivative of a transition count that quantifies a rate of change in a number of transitions detected within a time window between the behavioral regimes of the object;

comparing the derivative to one or more threshold values;

based on the derivative exceeding at least one of the one or more threshold values, generating a regime change alert; and

providing, by the device, the regime change alert and a portion of the video data for display as part of the regime change alert.

19 . The tangible, non-transitory, computer-readable medium as in claim 18 , wherein the portion of the video data is provided for display with an associated timestamp.

20 . The tangible, non-transitory, computer-readable medium as in claim 18 , wherein the portion of the video data depicts at least one transition between the behavioral regimes.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 21, 2022
From: LATAPIE, HUGO; KILIC, OZKAN; LAWRENCE, ADAM JAMES; LIU, GAOWEN; KOMPELLA, RAMANA RAO V. R.
To: CISCO TECHNOLOGY, INC.
Reel/Frame 061501/0220 →
Continuity (2)
Related Publication 20240135546A1 · Apr 25, 2024
Related Publication 20240233136A9 · Jul 11, 2024
References Cited (100)
US 10887197B2 · Fenoglio et al. · 2021 [cited by applicant]
US 10965516B2 · Fenoglio et al. · 2021 [cited by applicant]
US 11336506B1 · Li et al. · 2022 [cited by applicant]
US 20050180603A1 · Zoghlami et al. · 2005 [cited by applicant]
US 20080069482A1 · Komiya · 2008 [cited by applicant]
US 20100322516A1 · Xu et al. · 2010 [cited by applicant]
US 20110051992A1 · Cobb et al. · 2011 [cited by applicant]
US 20120063641A1 · Venkatesh et al. · 2012 [cited by applicant]
US 20130174116A1 · Pfeifer · 2013 [cited by applicant]
US 20140313330A1 · Carey · 2014 [cited by examiner]
US 20150186779A1 · Deng et al. · 2015 [cited by applicant]
US 20160105617A1 · Kirkby et al. · 2016 [cited by applicant]
US 20180033024A1 · Latapie · 2018 [cited by examiner]
US 20200251091A1 · Zhao · 2020 [cited by applicant]
US 20200319715A1 · Holz · 2020 [cited by applicant]
US 20210042532A1 · Latapie et al. · 2021 [cited by applicant]
US 20210125100A1 · Santos · 2021 [cited by examiner]
US 20210174155A1 · Smith et al. · 2021 [cited by applicant]
US 20210225409A1 · Lawlor · 2021 [cited by applicant]
US 20210258652A1 · Li et al. · 2021 [cited by applicant]
US 20210279615A1 · Latapie et al. · 2021 [cited by applicant]
US 20210312773A1 · Debnath et al. · 2021 [cited by applicant]
US 20210397849A1 · Lin et al. · 2021 [cited by applicant]
US 20220057519A1 · Goldstein · 2022 [cited by examiner]
US 20230154204A1 · Kahn · 2023 [cited by examiner]
WO WO2015027289A1 · 2015 [cited by applicant]
WO WO2019168323A1 · 2019 [cited by applicant]
WO WO2021251062A1 · 2021 [cited by applicant]
Agrawal, et al., “VQA: Visual Question Answering”, Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015, 25 pages, arXiv:1505.00468v7 [cs.CL]. [cited by applicant]
Ahmed, et al., “Reflection Detection in Image Sequences”, CVPR 2011, Jun./Jul. 2011, 9 pages, IEEE, Colorado Springs, CO. [cited by applicant]
Aleksander, Igor, “Machine consciousness” In Scholarpedia. 3(2):4162, Oct. 21, 2011, 7 pages. [cited by applicant]
Anderson, et al., “Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering”, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2018, pp. 6077-6086, IEEE, Salt Lake Cit… [cited by applicant]
Baudrillard, Jean, “Simulacra and Simulation”, 1981, 159 pages, Galilee. [cited by applicant]
Baz, et al., “Context-aware hybrid classification system for fine-grained retail product recognition”, 2016 IEEE 12th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP), Jul. 2016, 5 pages, IEEE, Bord… [cited by applicant]
Bělohlávek, Radim, “Concept lattices and order in fuzzy logic”, Annals of Pure and Applied Logic 128 (2004) 277-298, Elsevier. [cited by applicant]
Box, G. E. P., “Science and Statistics”, In Journal of the American Statistical Association, 71(356), Dec. 1976, pp. 791-799. [cited by applicant]
Chalmers, David J., “The Conscious Mind: In Search of a Fundamental Theory”, 1996, 433 pages, Oxford University Press, New York. [cited by applicant]
Chella, et al., “A cognitive framework for imitation learning”, Robotics and Autonomous Systems 54, Mar. 2006, pp. 403-408, Elsevier. [cited by applicant]
Chella, et al., “Artificial Consciousness”, Chapter 20, In Perception-Action Cycle, 2011, pp. 637-671, Springer, New York. [cited by applicant]
Chella, et al., “Machine Consciousness: A Manifesto for Robotics”, In International Journal of Machine Consciousness, 1(1), Jun. 2009, pp. 33-51, World Scientific Publishing Company. [cited by applicant]
Cohen, Paul R., “Projections as Concepts”, Computer Science Department Faculty Publication Series (194), https://scholarworks.umass.edu/cs/_faculty/_pubs/194, 1997, 6 pages, University of Massachusetts, Amherst. [cited by applicant]
Cui, et al., “A survey on network embedding”, IEEE Transactions on Knowledge and Data Engineering, vol. 31, Issue: 5, May 1, 2019, pp. 833-852, IEEE. [cited by applicant]
D'Amour, et al., “Underspecification Presents Challenges for Credibility in Modern Machine Learning”, Underspecification in Machine Learning, online: https://arxiv.org/pdf/2011.03395.pdf, Nov. 2020, 59 pages. [cited by applicant]
De Bono, Edward, “The Mechanism of Mind”, 1967, 276 pages, Penguin Books. [cited by applicant]
Düntsch, et al., “Modal-style operators in qualitative data analysis”, 2002 IEEE International Conference on Data Mining, 2002. Proceedings, Dec. 2002, pp. 155-162, IEEE, Maebashi City, Japan. [cited by applicant]
Franco, et al., “Grocery product detection and recognition”, Expert Systems With Applications 81 (2017), pp. 163-176, Elsevier Ltd. [cited by applicant]
George, et al., “Recognizing Products: A Per-exemplar Multi-label Image Classification Approach”, ECCV 2014, Part II, LNCS 8690, 2014, pp. 440-455, Springer International Publishing Switzerland. [cited by applicant]
Goertzel, et al., “CogPrime Architecture for Embodied Artificial General Intelligence”, 2013 IEEE Symposium on Computational Intelligence for Human-like Intelligence (CIHLI), Apr. 2013, pp. 60-67, IEEE, Singapore. [cited by applicant]
Goertzel, Ben, “OpenCogPrime: A Cognitive Synergy Based Architecture for Artificial General Intelligence”, 2009 8th IEEE International Conference on Cognitive Informatics, Jun. 2009, pp. 60-68, IEEE, Hong Kong, China. [cited by applicant]
Gorban, et al., “Blessing of dimensionality: mathematical foundations of the statistical physics of data”, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 376.2118, Ja… [cited by applicant]
Grover, et al., “node2vec: Scalable Feature Learning for Networks”, KDD '16: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Aug. 2016, pp. 855-864, Association for Co… [cited by applicant]
Hamilton, et al., “Representation Learning on Graphs: Methods and Applications”, Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 2017, 23 pages, IEEE. [cited by applicant]
Hammer, et al., “A Reasoning Based Model for Anomaly Detection in the Smart City Domain”, IntelliSys 2020, AISC 1251, pp. 144-159, 2021, Springer Nature Switzerland AG. [cited by applicant]
Hobbs, Jerry R., “Granularity”, In Proceedings of the Ninth International Joint Conference on Artificial Intelligence, 1985, pp. 432-435, Morgan Kaufmann. [cited by applicant]
Horowitz, Alexandra, “Smelling themselves: Dogs investigate their own odours longer when modified in an “olfactory mirror” test”, Behavioural Processes, 2017, 41 pages. [cited by applicant]
Johnson, Mark, “The Body in The Mind”, 1987, 268 pages, The University of Chicago Press. [cited by applicant]
Jacob, et al., “A Demonstration of the Exathlon Benchmarking Platform for Explainable Anomaly Detection”, Proceedings of the VLDB Endowment (PVLDB), Oct. 2021, 5 pages, HAL Open Science. [cited by applicant]
Kiryati, et al., “A probabilistic Hough transform”, Pattern Recognition. 24(4), 1991, pp. 303-316, The Pattern Recognition Society. [cited by applicant]
Korzybski, Alfred, “Manhood of Humanity, The Science and Art of Human Engineering”, 1921, 240 pages, E. P. Dutton & Company, New York, NY. [cited by applicant]
Korzybski, Alfred, “Science and Sanity: An Introduction to Non-Aristotelian Systems and General Semantics”, 5th Edition, 1994, 910 pages, Institute of General Semantics, New York, NY. [cited by applicant]
Korzybski, Alfred, “Videos—This Is Not That”, online: https://www.thisisnotthat.com/korzybski-videos/, accessed Nov. 18, 2021, 7 pages. [cited by applicant]
Lakoff, G., “Women, Fire, and Dangerous Things”, 1984, 631 pages, University of Chicago Press. [cited by applicant]
Latapie, et al., “A Metamodel and Framework for Artificial General Intelligence from Theory to Practice”, Journal of Artificial Intelligence and Consciousness, Feb. 12, 2021, 1:30, 24 pages, World Scientific Publishing … [cited by applicant]
Li, et al., “Concept learning via granular computing: A cognitive viewpoint”, Information Sciences 298 (2015), Published Dec. 2014, pp. 447-467, Elsevier Inc. [cited by applicant]
Lieto, et al., “Conceptual Spaces for Cognitive Architectures: A Lingua Franca for Different Levels of Representation”, Biologically Inspired Cognitive Architectures, May 19, 2017, 17 pages, Cognitive Robotics and Socia… [cited by applicant]
Lin, et al., “Progressive Mirror Detection”, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2020, 9 pages, IEEE, Seattle, WA. [cited by applicant]
Ma, et al., “Granular computing and Dual Galois Connection”, Information Sciences, 177(23), 2007, pp. 5365-5377, Elsevier Inc. [cited by applicant]
MacAulay, Thomas, “Facebook's chief AI scientist says GPT-3 is ‘not a very good’ Q&A system”, online: https://thenextweb.com/news/facebooks-yann-lecun-says-gpt-3-is-not-very-good-as-a-qa-or-dialog-system, Oct. 28, 2020,… [cited by applicant]
Murahari, et la., “Improving Generative Visual Dialog by Answering Diverse Questions”, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on… [cited by applicant]
Park, et al., “Identifying Reflected Images from Object Detector in Indoor Environment Utilizing Depth Information”, IEEE Robotics and Automation Letters, vol. 6, No. 2, Apr. 2021, pp. 635-642, IEEE. [cited by applicant]
Patel, et al., “Video Representation and Suspicious Event Detection Using Semantic Technologies”, online: http://semantic-web-journal.net/system/files/swj2427.pdf, Semantic Web 0, Sep. 10, 2020, accessed Aug. 9, 2021, 2… [cited by applicant]
Pauli, Wolfgang, “Part I. General: (A) theory. Some relations between electrochemical behaviour and the structure of colloids”, Jan. 1935, pp. 11-27, Transactions of the Faraday Society, vol. 1. [cited by applicant]
Scarselli, et al., “The Graph Neural Network Model”, IEEE Transactions on Neural Networks (vol. 20, Issue: 1, Jan. 2009), pp. 61-80, IEEE. [cited by applicant]
Speer, et al., “ConceptNet 5.5: An Open Multilingual Graph of General Knowledge”, online: https://arxiv.org/pdf/1612.03975.pdf, 2017, 9 pagers, Association for the Advancement of Artificial Intelligence. [cited by applicant]
Swanson, Bret, “The Exponential Internet”, online: https://www.uschamberfoundation.org/bhq/exponential-internet, accessed Nov. 19, 2021, 8 pages, The U.S. Chamber of Commerce Foundation. [cited by applicant]
Tan, et al., “EfficientDet: Scalable and Efficient Object Detection”, online: https://arxiv.org/pdf/1911.09070.pdf, Jul. 2020, 10 pages. [cited by applicant]
Taylor, J. G., “CODAM: A neural network model of consciousness”, Neural Networks 20 (2007), pp. 983-992, Elsevier Ltd. [cited by applicant]
Thórisson, et al., “Cumulative Learning”, Artificial General Intelligence—12th International Conference, AGI 2019, Proceedings, pp. 198-208, Springer. [cited by applicant]
Thórisson, Kristinn R., “A New Constructivist AI: From Manual Methods to Self-Constructive Systems”, Chapter 9, Apr. 2012, pp. 147-174, Atlantis Press Book. [cited by applicant]
Thórisson, Kristinn R., “Integrated AI Systems”, Minds & Machines, Mar. 17, 2007, pp. 11-25. [cited by applicant]
Tonioni, et al., “Product recognition in store shelves as a sub-graph isomorphism problem”, online: https://arxiv.org/abs/1707.08378, Sep. 2017, 14 pages. [cited by applicant]
Tripathy, et al., “Explaining Anomalies in Industrial Multivariate Time-Series Data with the Help of eXplainable AI”, 2022 IEEE International Conference on Big Data and Smart Computing (BigComp), Jan. 2022, 8 pages, IEE… [cited by applicant]
Unger, et al., “The Singular Universe and the Reality of Time: A Proposal in Natural Philosophy”, 2015, 558 pages, Cambridge University Press. [cited by applicant]
Unger, R. M. 2014. “Roberto Unger: Free Classical Social Theory from Illusions of False Necessity”, Online Lecture. 45 pages Retrieved on Nov. 22, 2021 from https://www.youtube.com/watch?v=yYOOwNRFTcY. [cited by applicant]
Wang, et al., “Concept Analysis via Rough Set and AFS Algebra”, Information Sciences 178 (2008), pp. 4125-4137, Elsevier Inc. [cited by applicant]
Wang, Pei, “Experience-grounded semantics: a theory for intelligent systems”, Aug. 2004, 33 pages, Elsevier Science. [cited by applicant]
Wang, Pei, “Insufficient Knowledge and Resources—A Biological Constraint and Its Functional Implications”, Biologically Inspired Cognitive Architectures II: Papers from the AAAI Fall Symposium (FS-09-01), 2009, pp. 188-… [cited by applicant]
Wang, Pei, “Non-axiomatic logic (nal) specification”, online: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.167.2069&rep=rep1&type=pdf, Oct. 2009, 88 pages. [cited by applicant]
Wang, Pei, “On Defining Artificial Intelligence”, Journal of Artificial General Intelligence 10(2) 2019, pp. 1-37, Sciendo. [cited by applicant]
Wang, et al. “Self in NARS, an AGI System”, vol. 5, Article 20, Mar. 2018, 15 pages, Frontiers in Robotics and AI. [cited by applicant]
Wang, et al., “SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems”, online: https://arxiv.org/pdf/1905.00537.pdf, 2019, 29 pages, 33rd Conference on Neural Information Processing Systems … [cited by applicant]
Wikipedia, “Wheat and chessboard problem”, online: https://en.wikipedia.org/wiki/Wheat_and_chessboard_problem, Oct. 2021, 5 pages, Wikimedia Foundation, Inc. [cited by applicant]
Wille, Rudolf, “Restructuring Lattice Theory: An Approach Based on Hierarchies of Concepts”, I. Rival (Ed.), Ordered Sets, 1982, pp. 314-339. [cited by applicant]
Xue, et al., “Real-Time Anomaly Detection and Feature Analysis Based on Time Series for Surveillance Video”, IEEE 5th International Conference on Universal Village ⋅ UV2020 ⋅ Session 3ABD-7, Oct. 2020, 7 pages, IEEE, Bo… [cited by applicant]
Yao, et al., “A Granular Computing Paradigm for Concept Learning”, Emerging Paradigms in Machine Learning, Springer, London, pp. 307-326, 2012. [cited by applicant]
Yao, Y. Y., “Information Granulation and Rough Set Approximation”, International Journal of Intelligent Systems, vol. 16, No. 1, 87-104, 2001. [cited by applicant]
Yao, Y. Y., “Integrative levels of granularity”, Human-Centric Information Processing Through Granular Modelling, 2009, 20 pages, Studies in Computational Intelligence, vol. 182. Springer, Berlin, Heidelberg. [cited by applicant]
Ying, et al., “Graph convolutional neural networks for web-scale recommender systems”, online: https://arxiv.org/pdf/1806.01973.pdf, In KDD '18: The 24th ACM SIGKDD International Conference on Knowledge Discovery & Data… [cited by applicant]
Zhou, et al., “Graph neural networks: A review of methods and applications”, AI Open, 2020, pp. 57-81, Elsevier B.V. [cited by applicant]
Zhu, et al., “Describing Unseen Videos via Multi-modal Cooperative Dialog Agents” Computer Vision—ECCV 2020, 17 pages, Lecture Notes in Computer Science, vol. 12368. Springer. [cited by applicant]