IP Library › Granted Patent US 12,382,001
Granted Patent B2
US 12,382,001 · App. 18/156,746 · Granted Aug 5, 2025

Event attendance monitoring using a virtual assistant

Inventors: Krishna Khadloya (San Jose, CA); Vaidhi Nathan (San Jose, CA); Chandan Gope (Cupertino, CA)
Assignee: Nice North America LLC
H04N7/183G06N20/00G06V10/764G06V20/10G06V20/52G06V20/70G06V40/10G06V40/103G06V40/174G10L17/00G10L25/51
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,382,001
App. No.
18/156,746
Granted
Aug 5, 2025
Kind
B2
Abstract

A function of a user-controlled virtual assistant (UCVA) device, such as a smart speaker, can be augmented using video or image information about an environment. In an example, a system for augmenting an UCVA device includes an image sensor configured to monitor an environment, a processor circuit configured to receive image information from the image sensor and use artificial intelligence to discern a presence of one or more known individuals in the environment from one or more other features in the environment. The system can include an interface coupled to the processor circuit and configured to provide identification information to the UCVA device about the one or more known human beings in the environment. The UCVA device can be configured by the identification information to update an operating mode of the UCVA device.

Claims (37)

1. An environment analysis device comprising:

a processor circuit configured to receive image information from an image sensor and to receive audio information from an audio sensor; and

a non-transitory memory circuit coupled to the processor circuit, the non-transitory memory circuit comprising instructions that, when performed by the processor circuit, configure the processor circuit to:

analyze at least one of the image information or the audio information to determine multiple individuals that are within a field of view of the image sensor, and utilize artificial intelligence to identify the multiple individuals present within the field of view at an event in an environment;

compare an expected attendance at the event in the environment with the identified multiple individuals present at the event in the environment to identify an absent individual; and

provide a notification to the absent individual about the event in the environment.

2. The environment analysis device of claim 1 , wherein the instruction configure the processor circuit to analyze the image information and the audio information together, using applied machine learning, to identify each of the individuals present at the event in the environment.

3. The environment analysis device of claim 1 , wherein the instructions further configure the processor circuit to:

analyze the at least one of the image information or the audio information to identify whether a specified individual is present in the environment; and

use the at least one of the image information and the audio information to confirm that the specified individual is present in the environment.

4. The environment analysis device of claim 3 , wherein the instructions further configure the processor circuit to perform a personalized task associated with the specified individual when the specified individual is confirmed to be present in the environment.

5. The environment analysis device of claim 3 , wherein the instructions further configure the processor circuit to determine a dwell time for the specified individual in the environment and determine the individual is unauthorized after a specified dwell duration elapses.

6. The environment analysis device of claim 3 , wherein the instructions to configure the processor circuit to compare the expected attendance at the event with the identified multiple individuals present at the event includes performing the comparison in response to confirming that the specified individual is present in the environment.

7. The environment analysis device of claim 6 , wherein the instructions further configure the processor circuit to, in response to confirming that the specified individual is present in the environment, cause the device to exit a security monitoring mode and enter an assistant mode, wherein in the security monitoring mode the device is configured to identify an adverse event in the environment, and in the assistant mode the device is configured to perform one or more tasks for the specified individual.

8. The environment analysis device of claim 1 , wherein the instructions configure the processor circuit to analyze the image information and the audio information together to identify the multiple individuals present at the event in the environment.

9. The environment analysis device of claim 1 , wherein the instructions further configure the processor circuit to:

use the at least one of the image information or the audio information to identify a particular individual, from among the multiple individuals, who is speaking at the event; and

record the image information and/or the audio information when the particular individual is speaking.

10. The environment analysis device of claim 1 , wherein the instructions further configure the processor circuit to identify a look direction of one or more of the identified individuals at the event in the environment.

11. The environment analysis device of claim 1 , wherein the instructions further configure the processor circuit to identify a mood of one or more of the identified individuals at the event in the environment.

12. A method comprising:

receiving, at a processor circuit of a virtual assistant device provided in an environment, respective signals with information about the environment, the signals provided by respective different environment sensors including an audio sensor and an image sensor; and

using the processor circuit of the virtual assistant device:

determining multiple individuals that are within a field of view of the image sensor;

applying artificial intelligence-based processing to analyze together the information about the environment as-received from the different environment sensors and, based on the analysis, identifying the multiple individuals present within the field of view at an event in the environment;

comparing an expected attendance at the event in the environment with the identified multiple individuals present at the event in the environment to identify an absent individual; and

providing a notification to the absent individual about the event in the environment.

13. The method of claim 12 , further comprising, using the processor circuit, identifying each of the multiple individuals present at the event using audio information from the audio sensor and using image information from the image sensor.

14. The method of claim 12 , further comprising, using the processor circuit:

analyzing at least one of audio information from the audio sensor and image information from the image sensor to identify whether a specified individual is present at the event in the environment; and

analyzing the at least one of the audio information from the audio sensor and the image information from the image sensor to confirm that the specified individual is present at the event in the environment.

15. The method of claim 14 , further comprising, using the processor circuit, performing a personalized task associated with the specified individual when the specified individual is confirmed to be present in the environment.

16. The method of claim 15 , wherein the personalized task includes the comparing the expected attendance at the event with the identified multiple individuals present at the event to identify the absent individual.

17. The method of claim 15 , further comprising, in response to confirming the specified individual is present, changing an operating mode of the virtual assistant device, including changing from a security monitoring mode to an assistant mode, wherein in the security monitoring mode the device is configured to identify an adverse event in the environment, and in the assistant mode the device is configured to perform one or more tasks for the specified individual.

18. The method of claim 14 , further comprising, using the processor circuit, determining a dwell time for the specified individual in the environment and determining the individual is unauthorized after a specified dwell time elapses.

19. The method of claim 12 , further comprising, using the processor circuit, identifying a particular individual, from among the identified individuals, who is speaking at the event and, in response, recording audio information from the audio sensor or image information from the image sensor when the particular individual is speaking.

20. The method of claim 19 , further comprising identifying the particular individual based on a determined mood or a determined look direction of one or more of the identified individuals at the event.

Assignments (2)
CHANGE OF NAME Recorded Jan 9, 2024
From: NORTEK SECURITY & CONTROL LLC
To: NICE NORTH AMERICA LLC
Reel/Frame 066242/0513 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 24, 2023
From: KHADLOYA, KRISHNA; NATHAN, VAIDHI; GOPE, CHANDAN
To: NORTEK SECURITY & CONTROL LLC
Reel/Frame 062473/0573 →
Continuity (8)
Continuation 17061193 · Oct 1, 2020
Continuation 16271183 · Feb 8, 2019
Provisional Application 62632421 · Feb 20, 2018
Provisional Application 62632409 · Feb 19, 2018
Provisional Application 62632410 · Feb 19, 2018
Provisional Application 62629029 · Feb 11, 2018
Provisional Application 62628148 · Feb 8, 2018
Related Publication 20230156162A1 · May 18, 2023
References Cited (105)
US 6028626A · Aviv · 2000 [cited by applicant]
US 6594629B1 · Basu et al. · 2003 [cited by applicant]
US 6681032B2 · Bortolussi et al. · 2004 [cited by applicant]
US 7113090B1 · Saylor et al. · 2006 [cited by applicant]
US 7847820B2 · Vallone et al. · 2010 [cited by applicant]
US 7999857B2 · Bunn et al. · 2011 [cited by applicant]
US 8139098B2 · Carter · 2012 [cited by applicant]
US 8237571B2 · Wang et al. · 2012 [cited by applicant]
US 8527278B2 · David · 2013 [cited by applicant]
US 8660249B2 · Rondeau et al. · 2014 [cited by applicant]
US 9105053B2 · Cao et al. · 2015 [cited by applicant]
US 9135797B2 · Couper et al. · 2015 [cited by applicant]
US 9208675B2 · Xu et al. · 2015 [cited by applicant]
US 9224044B1 · Laska et al. · 2015 [cited by applicant]
US 9230560B2 · Ehsani et al. · 2016 [cited by applicant]
US 9304736B1 · Whiteley et al. · 2016 [cited by applicant]
US 9313312B2 · Merrow et al. · 2016 [cited by applicant]
US 9479354B2 · Yamanishi et al. · 2016 [cited by applicant]
US 9740940B2 · Chattopadhyay et al. · 2017 [cited by applicant]
US 9830932B1 · Gunderson et al. · 2017 [cited by applicant]
US 9942518B1 · Tangeland · 2018 [cited by examiner]
US 10014003B2 · Kim et al. · 2018 [cited by applicant]
US 10032326B1 · Landers, Jr. · 2018 [cited by examiner]
US 10073428B2 · Bruhn et al. · 2018 [cited by applicant]
US 10469283B1 · Kinney et al. · 2019 [cited by applicant]
US 10569420B1 · Cohen · 2020 [cited by examiner]
US 10834365B2 · Khadloya et al. · 2020 [cited by applicant]
US 10978050B2 · Khadloya et al. · 2021 [cited by applicant]
US 20010047264A1 · Roundtree · 2001 [cited by applicant]
US 20050267605A1 · Lee et al. · 2005 [cited by applicant]
US 20060190419A1 · Bunn et al. · 2006 [cited by applicant]
US 20120026328A1 · Sethna et al. · 2012 [cited by applicant]
US 20120136689A1 · Ickman · 2012 [cited by examiner]
US 20120143363A1 · Liu et al. · 2012 [cited by applicant]
US 20120293606A1 · Watson · 2012 [cited by examiner]
US 20130237240A1 · Krantz · 2013 [cited by examiner]
US 20130275138A1 · Gruber et al. · 2013 [cited by applicant]
US 20140046878A1 · Lecomte et al. · 2014 [cited by applicant]
US 20140222436A1 · Binder et al. · 2014 [cited by applicant]
US 20150009278A1 · Modai · 2015 [cited by examiner]
US 20150149231A1 · Nicolas · 2015 [cited by examiner]
US 20150221321A1 · Christian · 2015 [cited by applicant]
US 20160364963A1 · Matsuoka et al. · 2016 [cited by applicant]
US 20160378861A1 · Eledath et al. · 2016 [cited by applicant]
US 20160379456A1 · Nongpiur et al. · 2016 [cited by applicant]
US 20170311053A1 · Ganjam · 2017 [cited by examiner]
US 20170329466A1 · Krenkler et al. · 2017 [cited by applicant]
US 20180047230A1 · Nye · 2018 [cited by examiner]
US 20180232902A1 · Albadawi · 2018 [cited by examiner]
US 20180261213A1 · Arik et al. · 2018 [cited by applicant]
US 20180342329A1 · Rufo et al. · 2018 [cited by applicant]
US 20190028759A1 · Yuan · 2019 [cited by examiner]
US 20190043525A1 · Huang et al. · 2019 [cited by applicant]
US 20190139565A1 · Chang et al. · 2019 [cited by applicant]
US 20190236554A1 · Hill · 2019 [cited by examiner]
US 20190246075A1 · Khadloya et al. · 2019 [cited by applicant]
US 20190259378A1 · Khadloya et al. · 2019 [cited by applicant]
US 20190362608A1 · Horling · 2019 [cited by applicant]
US 20190377325A1 · Angola Abreu · 2019 [cited by applicant]
US 20210029330A1 · Khadloya et al. · 2021 [cited by applicant]
US 20210210074A1 · Khadloya et al. · 2021 [cited by applicant]
CN 104346607 · 2014 [cited by applicant]
CN 106372576 · 2017 [cited by applicant]
CN 107609512 · 2018 [cited by applicant]
CN 108540762 · 2018 [cited by applicant]
U.S. Appl. No. 16/280,806 U.S. Pat. No. 10,978,050, filed Feb. 20, 2019, Audio Type Detection. [cited by applicant]
U.S. Appl. No. 17/203,269, filed Mar. 16, 2021, Audio Type Detection. [cited by applicant]
U.S. Appl. No. 16/271,183 U.S. Pat. No. 10,834,365, filed Feb. 8, 2019, Audio-Visual Monitoring Using a Virtual Assistant. [cited by applicant]
U.S. Appl. No. 17/061,193, filed Oct. 1, 2020, Audio-Visual Monitoring Using a Virtual Assistant. [cited by applicant]
“U.S. Appl. No. 17/203,269, Final Office Action mailed May 2, 2023”, 16 pgs. [cited by applicant]
“U.S. Appl. No. 17/203,269, Pre-Appeal Brief Conference Request filed Aug. 2, 2023”, 5 pgs. [cited by applicant]
“U.S. Appl. No. 17/203,269, Decision on Pre-Appeal Brief Request mailed Aug. 17, 2023”, 2 pgs. [cited by applicant]
“U.S. Appl. No. 17/203,269, Non Final Office Action mailed Dec. 6, 2023”, 20 pgs. [cited by applicant]
“U.S. Appl. No. 17/203,269, Response filed Feb. 22, 2024 to Non Final Office Action mailed Dec. 6, 2023”, 14 pgs. [cited by applicant]
Deller, John R, “Discrete-Time Processing of Speech Signals”, New York: Institute of Electrical and Electronics Engineers, (2015). [cited by applicant]
“Home Invasion: Google, Amazon Patent Filings Reveal Digital Home Assistant Privacy Problems”, Comsumer Watchdog Executive Summary, accessed Feb. 4, 2019, URL: https: www.consumerwatchdog.org sites default files Dec. 20… [cited by applicant]
“Eyewatch”, Eyewatch Website, URL: https: eye-watch.in Login.action, (accessed Feb. 4, 2019), 11 pgs. [cited by applicant]
“Home Automation with Smart Speakers—Amazon Echo vs. Google Home vs. Apple HomePod”, Citius Minds Blog, accessed Feb. 4, 2019, URL: https: www.citiusminds.com blog wp-content uploads 2017 08 Home_Automation_Smart_Speake… [cited by applicant]
“U.S. Appl. No. 16/271,183, Non Final Office Action mailed Dec. 11, 2019”, 18 pgs. [cited by applicant]
“U.S. Appl. No. 16/271,183, Response filed Mar. 10, 2020 to Non Final Office Action mailed Dec. 11, 2019”, 15 pgs. [cited by applicant]
“U.S. Appl. No. 16/280,806, Non Final Office Action mailed May 13, 2020”, 17 pgs. [cited by applicant]
“U.S. Appl. No. 16/271,183, Notice of Allowance mailed Jun. 25, 2020”, 6 pgs. [cited by applicant]
“U.S. Appl. No. 16/280,806, Response filed Sep. 14, 2020 to Non Final Office Action mailed May 13, 2020”, 18 pgs. [cited by applicant]
“U.S. Appl. No. 16/280,806, Notice of Allowance mailed Dec. 10, 2020”, 9 pgs. [cited by applicant]
“U.S. Appl. No. 17/061,193, Non Final Office Action mailed Dec. 10, 2020”, 14 pgs. [cited by applicant]
“U.S. Appl. No. 17/061,193, Response filed Mar. 10, 2021 to Non Final Office Action mailed Dec. 10, 2020”, 14 pgs. [cited by applicant]
“U.S. Appl. No. 17/061,193, Final Office Action mailed Jul. 2, 2021”, 17 pgs. [cited by applicant]
“U.S. Appl. No. 17/061,193, Response filed Aug. 27, 2021 to Final Office Action mailed Jul. 2, 2021”, 13 pgs. [cited by applicant]
“U.S. Appl. No. 17/061,193, Advisory Action mailed Sep. 22, 2021”, 2 pgs. [cited by applicant]
“U.S. Appl. No. 17/061,193, Non Final Office Action mailed Mar. 16, 2022”, 13 pgs. [cited by applicant]
“U.S. Appl. No. 17/061,193, Response filed Jul. 18, 2022 to Non Final Office Action mailed Mar. 16, 2022”, 14 pgs. [cited by applicant]
“U.S. Appl. No. 17/061,193, Final Office Action mailed Sep. 23, 2022”, 14 pgs. [cited by applicant]
“U.S. Appl. No. 17/203,269, Non Final Office Action mailed Dec. 9, 2022”, 16 pgs. [cited by applicant]
“U.S. Appl. No. 17/203,269, Response filed Apr. 10, 2023 to Non Final Office Action mailed Dec. 9, 2022”, 11 pgs. [cited by applicant]
Clavel, Chloe, “Events detection for an audio-based surveillance system”, 2005 IEEE International Conference on Multimedia and Expo. IEEE, (2005), 4 pgs. [cited by applicant]
Colangelo, Federico, “Enhancing audio surveillance with hierarchical recurrent neural networks”, 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). IEEE, (2017). [cited by applicant]
Dennis, Jonathan, “Spectrogram image feature for sound event classification in mismatched conditions”, IEEE signal processing letters 18.2, (2010), 130-133. [cited by applicant]
Huang, Hua, “WiDet: Wi-Fi Based Device-Free Passive Person Detection with Deep Convolutional Neural Networks”, Proc. of the 21st ACM Intl. Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems (… [cited by applicant]
Jacobson, Julie, “Patent: Vivint Door Knock Tech Might Help Identify Guests by Unique Sound Signatures”, CE Pro Magazine and Website, accessed Feb. 4, 2019, URL: https: www.cepro.com article patent_audio_analytics_ai_vi… [cited by applicant]
Jurrien, Ilse, “Samsung Bixby Smart Speaker met Camera en Display”, LetsGo Digital, accessed Feb. 4, 2019, w English machine translation, URL: https: nl.letsgodigital.org speakers-hifi samsung-bixby-smart-speaker , (Jun… [cited by applicant]
Metz, Rachel, “Using Deep Learning to Make Video Surveillance Smarter”, MIT Technology Review Online, [Online] Retrieved from the internet on Feb. 4, 2019: URL: https: www.technologyreview.com s 540396 using-deep-learni… [cited by applicant]
Ntalampiras, Stavros, “An adaptive framework for acoustic monitoring of potential hazards”, EURASIP Journal on Audio, Speech, and Music Processing 2009.1(2009): 594103, (2009), 15 pgs. [cited by applicant]
Rasti, Pejman, “Convolutional Neural Network Super Resolution for Face Recognition in Surveillance Monitoring”, Proc. of the Intl. Conference on Articulated Motion and Deformable Objects (AMDO), (Jul. 2, 2016), 175-184. [cited by applicant]
Shaw, C. Mitchell, “New Patents: Google Wants to Monitor Everything You and Your Children Do at Home”, The New American, accessed Feb. 4, 2019, URL: https: www.thenewamerican.com tech computers item 30733-new-patents-go… [cited by applicant]
Whitney, Lance, “How to Call Someone From Your Amazon Echo”, PC Magazine Online, accessed Feb. 4, 2019, URL: https: www.pcmag.com feature 358674 how-to-call-someone-from-your-amazon-echo, (Dec. 25, 2018), 13 pgs. [cited by applicant]
Cited By (1)
US 12,657,921