IP Library Granted Patent US 12,254,884
Granted Patent B2
US 12,254,884 · App. 18/421,254 · Granted Mar 18, 2025

Hotword detection on multiple devices

Inventor: Matthew Sharifi (Kilchberg, CH)
Assignee: Google LLC
G10L15/285G10L15/01G10L15/08G10L15/22G10L15/32G10L17/22G06F3/167G10L2015/088G10L2015/223
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,254,884
App. No.
18/421,254
Granted
Mar 18, 2025
Kind
B2
Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for hotword detection on multiple devices are disclosed. In one aspect, a method includes the actions of receiving, by a first computing device, audio data that corresponds to an utterance. The actions further include determining a first value corresponding to a likelihood that the utterance includes a hotword. The actions further include receiving a second value corresponding to a likelihood that the utterance includes the hotword, the second value being determined by a second computing device. The actions further include comparing the first value and the second value. The actions further include based on comparing the first value to the second value, initiating speech recognition processing on the audio data.

Claims (36)

1. A computer-implemented method executed on data processing hardware of a first computing device that causes the data processing hardware to perform operations comprising:

receiving audio data that corresponds to an utterance of a voice command and a hotword preceding the voice command, the utterance of the voice command and the hotword preceding the voice command captured by the first computing device and a second computing device, the first computing device and the second computing device each configured to respond to voice commands that are preceded by the hotword;

transmitting, from the first computing device to the second computing device after the second computing device captured the utterance of the hotword preceding the voice command, a message, the message received by the second computing device causing the second computing device to not respond to the voice command despite the second computing device capturing the utterance of the voice command and the hotword preceding the voice command; and

based on the message, causing the first computing device to respond to the voice command.

2. The computer-implemented method of claim 1 , wherein the message transmitted to the second computing device indicates an audio metric related to the utterance of the hotword captured by the first computing device.

3. The computer-implemented method of claim 2 , wherein the audio metric comprises a loudness of the utterance captured by the first computing device.

4. The computer-implemented method of claim 1 , wherein the operations further comprise, after receiving the audio data that corresponds to the utterance of the voice command and the hotword preceding the voice command:

receiving, from the second computing device, an additional message,

wherein causing the first computing device to respond to the voice command is further based on the additional message received from the second computing device.

5. The computer-implemented method of claim 4 , wherein the additional audio metric comprises a loudness of the utterance captured by the first computing device.

6. The computer-implemented method of claim 1 , wherein the operations further comprise, in response to receiving the audio data that corresponds to the utterance of the voice command and the hotword preceding the voice command, determining that the audio data likely includes the utterance of the hotword preceding the voice command without performing automated speech recognition on the audio data.

7. The computer-implemented method of claim 6 , wherein determining that the audio data likely includes the utterance of the hotword preceding the voice command comprises:

determining a hotword score that reflects a likelihood that the audio data includes the utterance of the hotword preceding the voice command; and

determining that the hotword score satisfies a threshold.

8. The computer-implemented method of claim 1 , wherein receiving the audio data comprises receiving the audio data while the first computing device is in a low power mode.

9. The computer-implemented method of claim 1 , wherein the first computing device and the second computing device are in communication via a local network.

10. The computer-implemented method of claim 1 , wherein the operations further comprise determining to perform speech recognition on the audio data.

11. A first computing device comprising:

data processing hardware; and

memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:

receiving audio data that corresponds to an utterance of a voice command and a hotword preceding the voice command, the utterance of the voice command and the hotword preceding the voice command captured by the first computing device and a second computing device, the first computing device and the second computing device each configured to respond to voice commands that are preceded by the hotword;

transmitting, from the first computing device to the second computing device after the second computing device captured the utterance of the hotword preceding the voice command, a message, the message received by the second computing device causing the second computing device to not respond to the voice command despite the second computing device capturing the utterance of the voice command and the hotword preceding the voice command; and

based on the message, causing the first computing device to respond to the voice command.

12. The first computing device of claim 11 , wherein the message transmitted to the second computing device indicates an audio metric related to the utterance of the hotword captured by the first computing device.

13. The first computing device of claim 12 , wherein the audio metric comprises a loudness of the utterance captured by the first computing device.

14. The first computing device of claim 11 , wherein the operations further comprise, after receiving the audio data that corresponds to the utterance of the voice command and the hotword preceding the voice command:

receiving, from the second computing device, an additional message,

wherein causing the first computing device to respond to the voice command is further based on the additional message received from the second computing device.

15. The first computing device of claim 14 , wherein the additional audio metric comprises a loudness of the utterance captured by the first computing device.

16. The first computing device of claim 11 , wherein the operations further comprise, in response to receiving the audio data that corresponds to the utterance of the voice command and the hotword preceding the voice command, determining that the audio data likely includes the utterance of the hotword preceding the voice command without performing automated speech recognition on the audio data.

17. The first computing device of claim 16 , wherein determining that the audio data likely includes the utterance of the hotword preceding the voice command comprises:

determining a hotword score that reflects a likelihood that the audio data includes the utterance of the hotword preceding the voice command; and

determining that the hotword score satisfies a threshold.

18. The first computing device of claim 11 , wherein receiving the audio data comprises receiving the audio data while the first computing device is in a low power mode.

19. The first computing device of claim 11 , wherein the first computing device and the second computing device are in communication via a local network.

20. The first computing device of claim 11 , wherein the operations further comprise determining to perform speech recognition on the audio data.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 24, 2024
From: SHARIFI, MATTHEW
To: GOOGLE INC.
Reel/Frame 066230/0345 →
CHANGE OF NAME Recorded Jan 24, 2024
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 066367/0361 →
Continuity (9)
Continuation 18150561 · Jan 5, 2023
Continuation 17137157 · Dec 29, 2020
Continuation 16553883 · Aug 28, 2019
Continuation 16171495 · Oct 26, 2018
Continuation 15346914 · Nov 9, 2016
Continuation 15088477 · Apr 1, 2016
Continuation 14675932 · Apr 1, 2015
Provisional Application 62061830 · Oct 9, 2014
Related Publication 20240169992A1 · May 23, 2024
References Cited (168)
US 4363102A · Holmgren et al. · 1982 [cited by applicant]
US 5455889A · Bahl et al. · 1995 [cited by applicant]
US 5659665A · Whelpley, Jr. · 1997 [cited by examiner]
US 5838384A · Schindler et al. · 1998 [cited by applicant]
US 5897616A · Kanevsky et al. · 1999 [cited by applicant]
US 6023676A · Erell · 2000 [cited by applicant]
US 6141644A · Kuhn et al. · 2000 [cited by applicant]
US 6567775B1 · Maali et al. · 2003 [cited by applicant]
US 6671672B1 · Heck · 2003 [cited by examiner]
US 6744860B1 · Schrage · 2004 [cited by examiner]
US 6826159B1 · Shaffer et al. · 2004 [cited by applicant]
US 6931375B1 · Bossemeyer, Jr. et al. · 2005 [cited by applicant]
US 6961704B1 · Phillips et al. · 2005 [cited by applicant]
US 6973426B1 · Schier et al. · 2005 [cited by applicant]
US 7016833B2 · Gable et al. · 2006 [cited by applicant]
US 7222072B2 · Chang · 2007 [cited by examiner]
US 7571014B1 · Lambourne et al. · 2009 [cited by applicant]
US 7720012B1 · Borah et al. · 2010 [cited by applicant]
US 7778816B2 · Reynar · 2010 [cited by applicant]
US 7904297B2 · Mirkovic · 2011 [cited by examiner]
US 8099288B2 · Zhang et al. · 2012 [cited by applicant]
US 8200488B2 · Kemp et al. · 2012 [cited by applicant]
US 8209174B2 · Al-Telmissani · 2012 [cited by applicant]
US 8214447B2 · Deslippe et al. · 2012 [cited by applicant]
US 8244534B2 · Qian et al. · 2012 [cited by applicant]
US 8340795B2 · Sherman · 2012 [cited by applicant]
US 8340975B1 · Rosenberger · 2012 [cited by examiner]
US 8588949B2 · Lambourne et al. · 2013 [cited by applicant]
US 8655653B2 · Vos · 2014 [cited by applicant]
US 8670985B2 · Lindahl et al. · 2014 [cited by applicant]
US 8713119B2 · Lindahl · 2014 [cited by applicant]
US 8717949B2 · Crinon et al. · 2014 [cited by applicant]
US 8719009B2 · Baldwin et al. · 2014 [cited by applicant]
US 8719018B2 · Dinerstein · 2014 [cited by applicant]
US 8768687B1 · Quasthoff · 2014 [cited by examiner]
US 8775191B1 · Sharifi et al. · 2014 [cited by applicant]
US 8805890B2 · Zhang et al. · 2014 [cited by applicant]
US 8838457B2 · Cerra et al. · 2014 [cited by applicant]
US 8938394B1 · Faaborg et al. · 2015 [cited by applicant]
US 8996372B1 · Secker-Walker · 2015 [cited by examiner]
US 9142218B2 · Schroeter · 2015 [cited by examiner]
US 9443527B1 · Watanabe et al. · 2016 [cited by applicant]
US 9916839B1 · Scalise et al. · 2018 [cited by applicant]
US 20020049596A1 · Burchard · 2002 [cited by examiner]
US 20020072905A1 · White · 2002 [cited by examiner]
US 20020123890A1 · Kopp · 2002 [cited by examiner]
US 20020193991A1 · Bennett · 2002 [cited by examiner]
US 20030200090A1 · Kawazoe · 2003 [cited by examiner]
US 20030220797A1 · Ito · 2003 [cited by applicant]
US 20030231746A1 · Hunter et al. · 2003 [cited by applicant]
US 20040078191A1 · Tian et al. · 2004 [cited by applicant]
US 20040101112A1 · Kuo · 2004 [cited by examiner]
US 20040158464A1 · Baker · 2004 [cited by applicant]
US 20040236570A1 · Tato et al. · 2004 [cited by applicant]
US 20050165607A1 · Di Fabbrizio · 2005 [cited by examiner]
US 20050171766A1 · Albesano et al. · 2005 [cited by applicant]
US 20060074656A1 · Mathias · 2006 [cited by examiner]
US 20060085188A1 · Goodwin et al. · 2006 [cited by applicant]
US 20060184370A1 · Kwak · 2006 [cited by examiner]
US 20070100620A1 · Tavares · 2007 [cited by examiner]
US 20070198262A1 · Mindlin et al. · 2007 [cited by applicant]
US 20080288256A1 · Agapi et al. · 2008 [cited by applicant]
US 20090240488A1 · White et al. · 2009 [cited by applicant]
US 20090258333A1 · Yu · 2009 [cited by examiner]
US 20090271190A1 · Niemisto et al. · 2009 [cited by applicant]
US 20090292541A1 · Daya · 2009 [cited by examiner]
US 20100070276A1 · Wasserblat · 2010 [cited by examiner]
US 20100110834A1 · Kim · 2010 [cited by examiner]
US 20110026722A1 · Jing · 2011 [cited by examiner]
US 20110054892A1 · Jung · 2011 [cited by examiner]
US 20110060587A1 · Phillips · 2011 [cited by examiner]
US 20110066429A1 · Shperling · 2011 [cited by examiner]
US 20110184730A1 · LeBeau et al. · 2011 [cited by applicant]
US 20110246181A1 · Liang et al. · 2011 [cited by applicant]
US 20110304648A1 · Kim et al. · 2011 [cited by applicant]
US 20120084087A1 · Yang et al. · 2012 [cited by applicant]
US 20120136923A1 · Grube · 2012 [cited by applicant]
US 20120232896A1 · Taleb · 2012 [cited by examiner]
US 20120245941A1 · Cheyer · 2012 [cited by applicant]
US 20120265528A1 · Gruber · 2012 [cited by examiner]
US 20130060571A1 · Soemo · 2013 [cited by examiner]
US 20130124207A1 · Sarin et al. · 2013 [cited by applicant]
US 20130132086A1 · Xu · 2013 [cited by examiner]
US 20130183944A1 · Mozer · 2013 [cited by examiner]
US 20130238326A1 · Kim et al. · 2013 [cited by applicant]
US 20130289996A1 · Fry · 2013 [cited by examiner]
US 20140006825A1 · Shenhav · 2014 [cited by applicant]
US 20140012573A1 · Hung · 2014 [cited by examiner]
US 20140012578A1 · Morioka · 2014 [cited by applicant]
US 20140088961A1 · Woodward · 2014 [cited by examiner]
US 20140156281A1 · Boyd · 2014 [cited by examiner]
US 20140163978A1 · Basye · 2014 [cited by examiner]
US 20140222430A1 · Rao · 2014 [cited by examiner]
US 20140229184A1 · Shires · 2014 [cited by examiner]
US 20140257821A1 · Adams · 2014 [cited by examiner]
US 20140278383A1 · Fan · 2014 [cited by examiner]
US 20140278435A1 · Ganong, III · 2014 [cited by examiner]
US 20150032238A1 · Ondo · 2015 [cited by examiner]
US 20150032443A1 · Karov et al. · 2015 [cited by applicant]
US 20150081295A1 · Yun et al. · 2015 [cited by applicant]
US 20150154953A1 · Bapat · 2015 [cited by examiner]
US 20150228274A1 · Leppanen et al. · 2015 [cited by applicant]
US 20150248885A1 · Koulomzin · 2015 [cited by applicant]
US 20150262577A1 · Nomura · 2015 [cited by examiner]
US 20150379989A1 · Balasubramanian et al. · 2015 [cited by applicant]
US 20160012465A1 · Sharp · 2016 [cited by applicant]
US 20160104483A1 · Foerster et al. · 2016 [cited by applicant]
US 20160155443A1 · Khan et al. · 2016 [cited by applicant]
US 20160189715A1 · Nishikawa · 2016 [cited by applicant]
US 20160217790A1 · Sharifi · 2016 [cited by applicant]
US 20160260431A1 · Newendorp et al. · 2016 [cited by applicant]
US 20160294973A1 · Bakshi et al. · 2016 [cited by applicant]
US 20170083285A1 · Meyers et al. · 2017 [cited by applicant]
US 20180061419A1 · Melendo Casado et al. · 2018 [cited by applicant]
US 20180182390A1 · Hughes et al. · 2018 [cited by applicant]
US 20180196776A1 · Hershko et al. · 2018 [cited by applicant]
US 20190304443A1 · Bhagwan · 2019 [cited by applicant]
JP S59180599A · 1984 [cited by applicant]
JP H0962293A · 1997 [cited by applicant]
JP H09127973A · 1997 [cited by applicant]
JP H1152976A · 1999 [cited by applicant]
JP H11231896A · 1999 [cited by applicant]
JP 2000075954A · 2000 [cited by applicant]
JP 2000310999A · 2000 [cited by applicant]
JP 2003223188A · 2003 [cited by applicant]
JP 2006227634A · 2006 [cited by applicant]
JP 2009104020A · 2009 [cited by applicant]
JP 2011232521A · 2011 [cited by applicant]
JP 201492777A · 2014 [cited by applicant]
JP 2015520409A · 2015 [cited by applicant]
JP 201921428A · 2019 [cited by applicant]
JP 2021184030A · 2021 [cited by applicant]
KR 1020140031391A · 2014 [cited by applicant]
WO 1998040875A1 · 1998 [cited by applicant]
WO 2013006385A1 · 2013 [cited by applicant]
WO 2013163113A1 · 2013 [cited by applicant]
WO 2014008194A1 · 2014 [cited by applicant]
WO 2014130463A2 · 2014 [cited by applicant]
WO 2015025330A1 · 2015 [cited by applicant]
Int'l Search Report and Written Opinion in Int'l Application No. PCT/US2015/052860, mailed Dec. 8, 2015, 12 pages. [cited by applicant]
Japanese Office Action issued in Japanese Application No. 2017-170185, mailed on Oct. 1, 2018, 17 pages (with English Abstract). [cited by applicant]
European Examination Report issued in European Application No. 162007288.8-1210, dated Aug. 1, 2018, 5 pages. [cited by applicant]
Notice of Allowance issued in U.S. Appl. No. 15/278,269, mailed on Jan. 12, 2018, 10 pages. [cited by applicant]
Japanese Office Action issued in Application No. 2016-250670, mailed on Mar. 26, 2018, 7 pages (English Translation). [cited by applicant]
Notice of Allowance in Japanese Application No. 2016-551250, mailed on Aug. 7, 2017, 3 pages (English translation). [cited by applicant]
Office Action issued in U.S. Appl. No. 15/278,269, mailed on Aug. 18, 2017, 23 pages. [cited by applicant]
Office Action issued in Korean Application No. 10-2016-7022701, mailed on Aug. 17, 2017, 4 pages (English translation). [cited by applicant]
Korean Office Action issued in Application No. 10-2016-7027999, mailed on Aug. 28, 2017, 9 pages (with English translation). [cited by applicant]
European Office Action issued in Application No. 15781811.3 mailed on Aug. 23, 2017, 6 pages. [cited by applicant]
Int'l Search Report and Written Opinion issued in Int'l Application No. PCT/US2017/045123, mailed on Nov. 8, 2017, 15 pages. [cited by applicant]
Korean Office Action issued in Application No. 10-2016-7033161, mailed on Dec. 19, 2017, 10 pages (with English translation). [cited by applicant]
Office Action in Japanese Application No. 2016-551250, mailed on Apr. 10, 2017, 6 pages (English translation). [cited by applicant]
Notice of Allowance in Korean Application No. 10-2016-7021778, mailed on Apr. 25, 2017, (English translation) 3 pages. [cited by applicant]
Notice of Allowance in U.S. Appl. No. 15/201,972 mailed on Jun. 1, 2017, 10 pages. [cited by applicant]
Extended European Search Report in European Application No. 16207288.8-1914, mailed on Apr. 4, 2017, 8 pages. [cited by applicant]
Office Action issued in Korean Application No. 10-2016-7027999, mailed Apr. 12, 2017, 9 pages (English translation). [cited by applicant]
Jae-Seung, Choi, “Text-dependent Speaker Recognition using Characteristic Vectors in Speech Signal and Normalized Recognition Method,” Journal of Korean Institute of Information Technology, 10(5), May 2012, 7 pages (Eng… [cited by applicant]
Auckenthaler et al. “Score Normalization for Text-independent Speaker Verification System,” Digital Signal Processing, vol. 10, 2000, 13 pages. [cited by applicant]
Int'l Preliminary Report on Patentability in Int'l Application No. PCT/US2015/052860, mailed on Apr. 11, 2017, 9 pages. [cited by applicant]
Int'l Preliminary Report on Patentability in Int'l Application No. PCT/US2015/030569 mailed on Jan. 24, 2017, 8 pages. [cited by applicant]
Office Action in Korean Application No. 10-2016-7021778 mailed on Dec. 13, 2016, 4 pages. [cited by applicant]
Extended European Search Report issued in Application No. 16195834.3-1910 mailed on Nov. 23, 2016, 9 pages. [cited by applicant]
Japanese Office Action issued in Application No. 2016-561322, mailed on Feb. 5, 2018, 6 pages (with English translation). [cited by applicant]
Japanese Office Action for the related Application No. 2019-91428 dated May 11, 2020. [cited by applicant]
Japanese Office Action for the related Japanese Patent Application No. 2021-184030, mailed on Feb. 6, 2023, 6 pages (English translation). [cited by applicant]
USPTO. Office Action relating to U.S. Appl. No. 17/242,273, dated Nov. 10, 2022. [cited by applicant]
USPTO. Office Action relating to U.S. Appl. No. 18/150,561, dated Aug. 17, 2023. [cited by applicant]
Japane Office action for the related application No. 2023-153929 dated Dec. 10, 2024. [cited by applicant]