IP Library › Granted Patent US 12,518,529
Granted Patent B2
US 12,518,529 · App. 17/987,557 · Granted Jan 6, 2026

Reflection detection in video analytics

Inventors: Hugo Latapie (Long Beach, CA); Ozkan Kilic (Long Beach, CA); Adam James Lawrence (Pasadena, CA); Gaowen Liu (Austin, TX); Ramana Rao V. R. Kompella (Cupertino, CA)
Assignee: Cisco Technology, Inc.
G06V20/40G06V20/52G06V40/10G06V2201/08
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,518,529
App. No.
17/987,557
Granted
Jan 6, 2026
Kind
B2
Abstract

In one embodiment, a device tracks objects in video data captured by one or more cameras in a location. The device represents spatial characteristics of the objects over time as timeseries. The device associates different portions of the timeseries with behavioral regimes of the objects. The device makes a determination that one of the objects is a reflection of another of the objects, based on a correlation between changes of their respective behavioral regimes.

Claims (49)

1 . A method comprising:

tracking, by a device, objects in video data captured by one or more cameras in a location;

computing, by the device, spatial characteristics of the objects;

generating, by the device, a timeseries of the spatial characteristics of the objects over time, wherein the timeseries represents behaviors of the objects;

associating, by the device, different portions of the timeseries with behavioral regimes of the objects, wherein each behavioral regime is defined by a distinctive pattern in the timeseries indicative of an activity performed by the objects;

computing, by the device, a correlation between each behavioral regime of different objects based on their respective timeseries; and

making, by the device and based on the correlation, a determination that one of the objects is a reflection of another of the objects.

2 . The method as in claim 1 , wherein the spatial characteristics comprise detected centroids of the objects.

3 . The method as in claim 1 , wherein the behavioral regimes comprise different patterns of the timeseries associated with the objects performing various actions.

4 . The method as in claim 1 , wherein the objects are people in the location.

5 . The method as in claim 1 , further comprising:

identifying, by the device and based on the determination, a region of the video data as depicting a reflective surface in the location; and

stopping, by the device, tracking objects in the region.

6 . The method as in claim 5 , wherein the device identifies the region in part by:

computing a heatmap of reflections detected in the region.

7 . The method as in claim 1 , wherein the determination is based in part on an amount of mutual information between the timeseries.

8 . The method as in claim 1 , wherein the objects comprise vehicles.

9 . The method as in claim 1 , further comprising:

providing, by the device, an indication of the determination to a user interface for presentation to a user.

10 . The method as in claim 1 , wherein the device is an edge device in a network.

11 . An apparatus, comprising:

a network interface to communicate with a computer network;

a processor coupled to the network interface and configured to execute one or more processes; and

a memory configured to store a process that is executed by the processor, the process when executed configured to:

track objects in video data captured by one or more cameras in a location;

compute spatial characteristics of the objects;

generate a timeseries of the spatial characteristics of the objects over time, wherein the timeseries represents behaviors of the objects;

associate different portions of the timeseries with behavioral regimes of the objects, wherein each behavioral regime is defined by a distinctive pattern in the timeseries indicative of an activity performed by the objects;

compute a correlation between each behavioral regime of different objects based on their respective timeseries; and

make a determination, based on the correlation, that one of the objects is a reflection of another of the objects.

12 . The apparatus as in claim 11 , wherein the spatial characteristics comprise detected centroids of the objects.

13 . The apparatus as in claim 11 , wherein the behavioral regimes comprise different patterns of the timeseries associated with the objects performing various actions.

14 . The apparatus as in claim 11 , wherein the objects are people in the location.

15 . The apparatus as in claim 11 , wherein the process when executed is further configured to:

identify, based on the determination, a region of the video data as depicting a reflective surface in the location; and

stop tracking objects in the region.

16 . The apparatus as in claim 15 , wherein the apparatus identifies the region in part by:

computing a heatmap of reflections detected in the region.

17 . The apparatus as in claim 11 , wherein the determination is based in part on an amount of mutual information between the timeseries.

18 . The apparatus as in claim 11 , wherein the objects comprise vehicles.

19 . The apparatus as in claim 11 , wherein the process when executed is further configured to:

provide an indication of the determination to a user interface for presentation to a user.

20 . A tangible, non-transitory, computer-readable medium storing program instructions that cause a device to execute a process comprising:

tracking, by the device, objects in video data captured by one or more cameras in a location;

computing, by the device, spatial characteristics of the objects;

generating, by the device, a timeseries of the spatial characteristics of the objects over time, wherein the timeseries represents behaviors of the objects;

associating, by the device, different portions of the timeseries with behavioral regimes of the objects, wherein each behavioral regime is defined by a distinctive pattern in the timeseries indicative of an activity performed by the objects;

computing, by the device, a correlation between each behavioral regime of different objects based on their respective timeseries; and

making, by the device and based on the correlation, a determination that one of the objects is a reflection of another of the objects.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 15, 2022
From: LATAPIE, HUGO; KILIC, OZKAN; LAWRENCE, ADAM JAMES; LIU, GAOWEN; KOMPELLA, RAMANA RAO V. R.
To: CISCO TECHNOLOGY, INC.
Reel/Frame 061781/0312 →
Continuity (1)
Related Publication 20240161496A1 · May 16, 2024
References Cited (109)
US 10887197B2 · Fenoglio et al. · 2021 [cited by applicant]
US 10965516B2 · Fenoglio et al. · 2021 [cited by applicant]
US 11336506B1 · Li et al. · 2022 [cited by applicant]
US 11594043B1 · Xu · 2023 [cited by examiner]
US 12062367B1 · Huynh · 2024 [cited by examiner]
US 20050180603A1 · Zoghlami et al. · 2005 [cited by applicant]
US 20080069482A1 · Komiya · 2008 [cited by applicant]
US 20090153661A1 · Cheng · 2009 [cited by examiner]
US 20100322516A1 · Xu et al. · 2010 [cited by applicant]
US 20110051992A1 · Cobb et al. · 2011 [cited by applicant]
US 20120063641A1 · Venkatesh et al. · 2012 [cited by applicant]
US 20130174116A1 · Pfeifer · 2013 [cited by applicant]
US 20150186779A1 · Deng et al. · 2015 [cited by applicant]
US 20160105617A1 · Kirkby et al. · 2016 [cited by applicant]
US 20160328613A1 · Gaidon et al. · 2016 [cited by applicant]
US 20190073538A1 · Ashani · 2019 [cited by examiner]
US 20200251091A1 · Zhao · 2020 [cited by applicant]
US 20200319715A1 · Holz · 2020 [cited by applicant]
US 20210042532A1 · Latapie et al. · 2021 [cited by applicant]
US 20210174155A1 · Smith et al. · 2021 [cited by applicant]
US 20210225409A1 · Lawlor · 2021 [cited by applicant]
US 20210258652A1 · Li et al. · 2021 [cited by applicant]
US 20210279615A1 · Latapie et al. · 2021 [cited by applicant]
US 20210312773A1 · Debnath et al. · 2021 [cited by applicant]
US 20210390423A1 · Latapie et al. · 2021 [cited by applicant]
US 20210397849A1 · Lin et al. · 2021 [cited by applicant]
US 20220101654A1 · Zhang · 2022 [cited by examiner]
US 20220138509A1 · Crosby et al. · 2022 [cited by applicant]
US 20230008567A1 · Zikos · 2023 [cited by examiner]
US 20240087282A1 · Young · 2024 [cited by examiner]
WO WO2015027289A1 · 2015 [cited by applicant]
WO WO2019168323A1 · 2019 [cited by applicant]
WO WO2021251062A1 · 2021 [cited by applicant]
Guan, Huankang, Jiaying Lin, and Rynson WH Lau. “Learning semantic associations for mirror detection.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. (Year: 2022). [cited by examiner]
Li Y, Brown MS. Exploiting reflection change for automatic reflection removal. InProceedings of the IEEE international conference on computer vision 2013 (pp. 2432-2439). (Year: 2013). [cited by examiner]
Amanlou A, Suratgar AA, Tavoosi J, Mohammadzadeh A, Mosavi A. Single-image reflection removal using deep learning: a systematic review. IEEE Access. Mar. 2, 2022;10:29937-53. (Year: 2022). [cited by examiner]
Agrawal, et al., “VQA: Visual Question Answering”, Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015, 25 pages, arXiv:1505.00468v7 [cs.CL]. [cited by applicant]
Ahmed, et al., “Reflection Detection in Image Sequences”, CVPR 2011, Jun./Jul. 2011, 9 pages, IEEE, Colorado Springs, CO. [cited by applicant]
Aleksander, Igor, “Machine consciousness” In Scholarpedia. 3(2):4162, Oct. 21, 2011, 7 pages. [cited by applicant]
Anderson, et al., “Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering”, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2018, pp. 6077-6086, IEEE, Salt Lake Cit… [cited by applicant]
Baudrillard, Jean, “Simulacra and Simulation”, 1981, 159 pages, Galilee. [cited by applicant]
Baz, et al., “Context-aware hybrid classification system for fine-grained retail product recognition”, 2016 IEEE 12th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP), Jul. 2016, 5 pages, IEEE, Bord… [cited by applicant]
Bělohlávek, Radim, “Concept lattices and order in fuzzy logic”, Annals of Pure and Applied Logic 128 (2004) 277-298, Elsevier. [cited by applicant]
Box, G. E. P., “Science and Statistics”, In Journal of the American Statistical Association, 71(356), Dec. 1976, pp. 791-799. [cited by applicant]
Chalmers, David J., “The Conscious Mind: In Search of a Fundamental Theory”, 1996, 433 pages, Oxford University Press, New York. [cited by applicant]
Chella, et al., “A cognitive framework for imitation learning”, Robotics and Autonomous Systems 54, Mar. 2006, pp. 403-408, Elsevier. [cited by applicant]
Chella, et al., “Artificial Consciousness”, Chapter 20, In Perception-Action Cycle, 2011, pp. 637-671, Springer, New York. [cited by applicant]
Chella, et al., “Machine Consciousness: A Manifesto for Robotics”, In International Journal of Machine Consciousness, 1(1), Jun. 2009, pp. 33-51, World Scientific Publishing Company. [cited by applicant]
Cohen, Paul R., “Projections as Concepts”, Computer Science Department Faculty Publication Series (194), https://scholarworks.umass.edu/cs/_faculty/_pubs/194, 1997, 6 pages, University of Massachusetts, Amherst. [cited by applicant]
Cui, et al., “A survey on network embedding”, IEEE Transactions on Knowledge and Data Engineering, vol. 31, Issue: 5, May 1, 2019, pp. 833-852, IEEE. [cited by applicant]
D'Amour, et al., “Underspecification Presents Challenges for Credibility in Modern Machine Learning”, Underspecification in Machine Learning, online: https://arxiv.org/pdf/2011.03395.pdf, Nov. 2020, 59 pages. [cited by applicant]
De Bono, Edward, “The Mechanism of Mind”, 1967, 276 pages, Penguin Books. [cited by applicant]
Düntsch, et al., “Modal-style operators in qualitative data analysis”, 2002 IEEE International Conference on Data Mining, 2002. Proceedings, Dec. 2002, pp. 155-162, IEEE, Maebashi City, Japan. [cited by applicant]
Franco, et al., “Grocery product detection and recognition”, Expert Systems With Applications 81 (2017), pp. 163-176, Elsevier Ltd. [cited by applicant]
George, et al., “Recognizing Products: A Per-exemplar Multi-label Image Classification Approach”, ECCV 2014, Part II, LNCS 8690, 2014, pp. 440-455, Springer International Publishing Switzerland. [cited by applicant]
Goertzel, et al., “CogPrime Architecture for Embodied Artificial General Intelligence”, 2013 IEEE Symposium on Computational Intelligence for Human-like Intelligence (CIHLI), Apr. 2013, pp. 60-67, IEEE, Singapore. [cited by applicant]
Goertzel, Ben, “OpenCogPrime: A Cognitive Synergy Based Architecture for Artificial General Intelligence”, 2009 8th IEEE International Conference on Cognitive Informatics, Jun. 2009, pp. 60-68, IEEE, Hong Kong, China. [cited by applicant]
Gorban, et al., “Blessing of dimensionality: mathematical foundations of the statistical physics of data”, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 376.2118, Ja… [cited by applicant]
Grover, et al., “node2vec: Scalable Feature Learning for Networks”, KDD '16: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Aug. 2016, pp. 855-864, Association for Co… [cited by applicant]
Hamilton, et al., “Representation Learning on Graphs: Methods and Applications”, Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 2017, 23 pages, IEEE. [cited by applicant]
Hammer, et al., “A Reasoning Based Model for Anomaly Detection in the Smart City Domain”, IntelliSys 2020, AISC 1251, pp. 144-159, 2021, Springer Nature Switzerland AG. [cited by applicant]
Hobbs, Jerry R., “Granularity”, In Proceedings of the Ninth International Joint Conference on Artificial Intelligence, 1985, pp. 432-435, Morgan Kaufmann. [cited by applicant]
Horowitz, Alexandra, “Smelling themselves: Dogs investigate their own odours longer when modified in an “olfactory mirror” test”, Behavioural Processes, 2017, 41 pages. [cited by applicant]
Johnson, Mark, “The Body in The Mind”, 1987, 268 pages, The University of Chicago Press. [cited by applicant]
Jacob, et al., “A Demonstration of the Exathlon Benchmarking Platform for Explainable Anomaly Detection”, Proceedings of the VLDB Endowment (PVLDB), Oct. 2021, 5 pages, HAL Open Science. [cited by applicant]
Jawed, et al., “Self-Supervised Learning for Semi-Supervised Time Series Classification”, Pacific-Asia Conference on Knowledge Discovery and Data Mining, PAKDD 2020: Advances in Knowledge Discovery and Data Mining, May … [cited by applicant]
Kiryati, et al., “A probabilistic Hough transform”, Pattern Recognition. 24(4), 1991, pp. 303-316, The Pattern Recognition Society. [cited by applicant]
Korzybski, Alfred, “Manhood of Humanity, The Science and Art of Human Engineering”, 1921, 240 pages, E. P. Dutton & Company, New York, NY. [cited by applicant]
Korzybski, Alfred, “Science and Sanity: An Introduction to Non-Aristotelian Systems and General Semantics”, 5th Edition, 1994, 910 pages, Institute of General Semantics, New York, NY. [cited by applicant]
Korzybski, Alfred, “Videos—This Is Not That”, online: https://www.thisisnotthat.com/korzybski-videos/, accessed Nov. 18, 2021, 7 pages. [cited by applicant]
Lakoff, G., “Women, Fire, and Dangerous Things”, 1984, 631 pages, University of Chicago Press. [cited by applicant]
Latapie, et al., “A Metamodel and Framework for Artificial General Intelligence from Theory to Practice”, Journal of Artificial Intelligence and Consciousness, Feb. 12, 2021, 1:30, 24 pages, World Scientific Publishing … [cited by applicant]
Li, et al., “Concept learning via granular computing: A cognitive viewpoint”, Information Sciences 298 (2015), Published Dec. 2014, pp. 447-467, Elsevier Inc. [cited by applicant]
Lieto, et al., “Conceptual Spaces for Cognitive Architectures: A Lingua Franca for Different Levels of Representation”, Biologically Inspired Cognitive Architectures 19, May 2017, 17 pages, Cognitive Robotics and Social… [cited by applicant]
Lin, et al., “Progressive Mirror Detection”, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2020, 9 pages, IEEE, Seattle, WA. [cited by applicant]
Ma, et al., “Granular computing and Dual Galois Connection”, Information Sciences, 177(23), 2007, pp. 5365-5377, Elsevier Inc. [cited by applicant]
Macaulay, Thomas, “Facebook's chief Al scientist says GPT-3 is ‘not a very good’ Q&A system”, online: https://thenextweb.com/news/facebooks-yann-lecun-says-gpt-3-is-not-very-good-as-a-qa-or-dialog-system, Oct. 28, 2020,… [cited by applicant]
Murahari, et la., “Improving Generative Visual Dialog by Answering Diverse Questions”, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on… [cited by applicant]
Park, et al., “Identifying Reflected Images from Object Detector in Indoor Environment Utilizing Depth Information”, IEEE Robotics and Automation Letters, vol. 6, No. 2, Apr. 2021, pp. 635-642, IEEE. [cited by applicant]
Patel, et al., “Video Representation and Suspicious Event Detection Using Semantic Technologies”, online: http://semantic-web-journal.net/system/files/swj2427.pdf, Semantic Web 0, Sep. 10, 2020, accessed Aug. 9, 2021, 2… [cited by applicant]
Pauli, Wolfgang, “Part I. General: (A) theory. Some relations between electrochemical behaviour and the structure of colloids”, Jan. 1935, pp. 11-27, Transactions of the Faraday Society, vol. 1. [cited by applicant]
Scarselli, et al., “The Graph Neural Network Model”, IEEE Transactions on Neural Networks (vol. 20, Issue: 1, Jan. 2009), pp. 61-80, IEEE. [cited by applicant]
Speer, et al., “ConceptNet 5.5: An Open Multilingual Graph of General Knowledge”, online: https://arxiv.org/pdf/1612.03975.pdf, 2017, 9 pagers, Association for the Advancement of Artificial Intelligence. [cited by applicant]
Swanson, Bret, “The Exponential Internet”, online: https://www.uschamberfoundation.org/bhq/exponential-internet, accessed Nov. 19, 2021, 8 pages, The U.S. Chamber of Commerce Foundation. [cited by applicant]
Tan, et al., “EfficientDet: Scalable and Efficient Object Detection”, online: https://arxiv.org/pdf/1911.09070.pdf, Jul. 2020, 10 pages. [cited by applicant]
Taylor, J. G., “Codam: A neural network model of consciousness”, Neural Networks 20 (2007), pp. 983-992, Elsevier Ltd. [cited by applicant]
Thorisson, et al., “Cumulative Learning”, Artificial General Intelligence—12th International Conference, AGI 2019, Proceedings, pp. 198-208, Springer. [cited by applicant]
Thorisson, Kristinn R., “A New Constructivist AI: From Manual Methods to Self-Constructive Systems”, Chapter 9, Apr. 2012, pp. 147-174, Atlantis Press Book. [cited by applicant]
Thorisson, Kristinn R., “Integrated AI Systems”, Minds & Machines 17, Mar. 2007, pp. 11-25. [cited by applicant]
Tonioni, et al., “Product recognition in store shelves as a sub-graph isomorphism problem”, online: https://arxiv.org/abs/1707.08378, Sep. 2017, 14 pages. [cited by applicant]
Tripathy, et al., “Explaining Anomalies in Industrial Multivariate Time-Series Data with the Help of explainable AI”, 2022 IEEE International Conference on Big Data and Smart Computing (BigComp), Jan. 2022, 8 pages, IEE… [cited by applicant]
Unger, et al., “The Singular Universe and the Reality of Time: A Proposal in Natural Philosophy”, 2015, 558 pages, Cambridge University Press. [cited by applicant]
Unger, R. M. 2014. “Roberto Unger: Free Classical Social Theory from Illusions of False Necessity”, Online Lecture. 45 pages Retrieved on Nov. 22, 2021 from https://www.youtube.com/watch?v=yYOOwNRFTcY. [cited by applicant]
Wang, et al., “Concept Analysis via Rough Set and AFS Algebra”, Information Sciences 178 (2008), pp. 4125-4137, Elsevier Inc. [cited by applicant]
Wang, Pei, “Experience-grounded semantics: a theory for intelligent systems”, Aug. 2004, 33 pages, Elsevier Science. [cited by applicant]
Wang, Pei, “Insufficient Knowledge and Resources—A Biological Constraint and Its Functional Implications”, Biologically Inspired Cognitive Architectures II: Papers from the AAAI Fall Symposium (FS-09-01), 2009, pp. 188-… [cited by applicant]
Wang, Pei, “Non-axiomatic logic (nal) specification”, online: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.167.2069&rep=rep1&type=pdf, Oct. 2009, 88 pages. [cited by applicant]
Wang, Pei, “On Defining Artificial Intelligence”, Journal of Artificial General Intelligence 10(2) 2019, pp. 1-37, Sciendo. [cited by applicant]
Wang, et al. “Self in NARS, an AGI System”, vol. 5, Article 20, Mar. 2018, 15 pages, Frontiers in Robotics and AI. [cited by applicant]
Wang, et al., “SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems”, online: https://arxiv.org/pdf/1905.00537.pdf, 2019, 29 pages, 33rd Conference on Neural Information Processing Systems … [cited by applicant]
Wikipedia, “Wheat and chessboard problem”, online: https://en.wikipedia.org/wiki/Wheat_and_chessboard_problem, Oct. 2021, 5 pages, Wikimedia Foundation, Inc. [cited by applicant]
Wille, Rudolf, “Restructuring Lattice Theory: An Approach Based on Hierarchies of Concepts”, I. Rival (Ed.), Ordered Sets, 1982, pp. 314-339. [cited by applicant]
Xue, et al., “Real-Time Anomaly Detection and Feature Analysis Based on Time Series for Surveillance Video”, IEEE 5th International Conference on Universal Village 108 UV2020 ⋅ Session 3ABD-7, Oct. 2020, 7 pages, IEEE,… [cited by applicant]
Yao, et al., “A Granular Computing Paradigm for Concept Learning”, Emerging Paradigms in Machine Learning, Springer, London, pp. 307-326, 2012. [cited by applicant]
Yao, Y. Y., “Information Granulation and Rough Set Approximation”, International Journal of Intelligent Systems, vol. 16, No. 1, 87-104, 2001. [cited by applicant]
Yao, Y. Y., “Integrative levels of granularity”, Human-Centric Information Processing Through Granular Modelling, 2009, 20 pages, Studies in Computational Intelligence, vol. 182. Springer, Berlin, Heidelberg. [cited by applicant]
Ying, et al., “Graph convolutional neural networks for web-scale recommender systems”, online: https://arxiv.org/pdf/1806.01973.pdf, In KDD '18: The 24th ACM SIGKDD International Conference on Knowledge Discovery & Data… [cited by applicant]
Zhou, et al., “Graph neural networks: A review of methods and applications”, AI Open, 2020, pp. 57-81, Elsevier B.V. [cited by applicant]
Zhu, et al., “Describing Unseen Videos via Multi-modal Cooperative Dialog Agents” Computer Vision—ECCV 2020, 17 pages, Lecture Notes in Computer Science, vol. 12368. Springer. [cited by applicant]