IP Library › Granted Patent US 12,387,715
Granted Patent B2
US 12,387,715 · App. 18/380,538 · Granted Aug 12, 2025

Learning offline voice commands based on usage of online voice commands

Inventors: Vikram Aggarwal (Palo Alto, CA); Moises Morgenstern Gali (San Francisco, CA)
Assignee: GOOGLE LLC
G10L15/07G10L15/19G10L15/22G10L15/30G10L15/32G10L2015/228
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,387,715
App. No.
18/380,538
Filed
Oct 16, 2023
Granted
Aug 12, 2025
Kind
B2
Art Unit
2658
USPC
704/275
Abstract

Methods, systems, apparatus, including computer programs encoded on a computer storage medium, for a user device to learn offline voice actions. In one aspect, the method includes actions of detecting, by the user device, an utterance at a first time when the user device is connected to a server by a network, providing, by the user device, the utterance to the server using the network, receiving, by the user device and from the server, an update to the grammar of the user device, detecting, by the user device, a subsequent utterance of the utterance at a second time when the user device is not connected to the server by a network, and in response to detecting, by the user device, the subsequent utterance of the utterance at the second time, identifying, by the user device, an operation to perform based on (i) the subsequent utterance, and (ii) the updated grammar.

Claims (69)

1. A method comprising:

receiving a transcription of an utterance that was detected via a first user interface input, wherein the transcription of the utterance includes one or more terms corresponding to the utterance;

identifying, based on the terms of the transcription of the utterance, an operation to be performed responsive to the utterance, wherein the operation to be performed responsive to the utterance comprises changing a state of a user device comprising the first user interface;

determining, whether the operation can be performed at the user device responsive to the utterance and without requesting information from a server, wherein determining that the operation can be performed by the user device responsive to the utterance and without requesting information from the server comprises:

determining that the user device does not require data from the server to change the state responsive to the utterance, and

determining that the user device includes one or more rules associated with changing the state responsive to the utterance; and

based on determining, that the operation can be performed by the user device responsive to the utterance and without requesting information from the server:

causing, based on the one or more terms of the transcription corresponding to the utterance and one or more of the rules associated with the operation to be performed responsive to the utterance, an update to a grammar stored locally on the user device to be provided,

wherein, when it is determined, that the operation cannot be performed by the user device responsive to the utterance and without requesting information from the server, no update is provided to the grammar stored locally on the user device;

subsequent to, and in response to, causing the update to the grammar stored locally on the user device to be provided:

identifying a user profile that is associated with the user device, wherein identifying the user profile that is associated with the user device comprises identifying the user profile based on the user profile being associated with the user device and the user device being the user device to which the update to the grammar is provided;

identifying, based on the user profile, one or more other user devices that are registered to a user of the user device, wherein identifying the one or more other user devices based on the user profile is in response to the user profile being identified as associated with the user device;

determining, whether the operation can be performed by one or more of the other user devices responsive to the utterance and without requesting information from the server; and

causing, based on the identification of an other user device of the other user devices that are registered to the user of the user device, based on determining the operation can be performed by the other user device responsive to the utterance and without requesting information from the server, and based on the one or more terms, an update to an additional grammar stored locally on the other user device,

wherein, when it is determined, that the operation cannot be performed by the other user device responsive to the utterance and without requesting information from the server, no update is provided to the additional grammar stored locally on the other user device.

2. The method of claim 1 , wherein causing the update to the grammar stored locally on the user device to be provided includes causing provision of the one or more terms to the user device.

3. The method of claim 2 , further comprising:

associating the one or more terms with a predetermined time period, wherein the predetermined time period indicates a maximum period of time the one or more terms will be included in the grammar absent receiving a subsequent utterance of the one or more terms by the user to perform the operation.

4. The method of claim 1 , further comprising:

causing generation, by the server, of a data object that, when executed, causes the user device to initiate performance of the operation responsive to the utterance.

5. The method of claim 4 , wherein causing an update to the grammar stored locally on the user device to be provided further comprises causing provision of the generated data object to the user device.

6. The method of claim 1 , wherein the update to the grammar enables the user device to execute a data object that initiates performance of the operation in response to subsequently receiving the utterance, and without requesting information from the server.

7. The method of claim 1 , wherein the other user devices comprise one of: a smartphone, a smartwatch, a laptop computer, a tablet computer, a desktop computer, a smart television, a set top box, a smart speaker, and/or a digital home assistant.

8. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

receiving a transcription of an utterance that was detected via a first user interface input, wherein the transcription of the utterance includes one or more terms corresponding to the utterance;

identifying, based on the terms of the transcription of the utterance, an operation to be performed responsive to the utterance, wherein the operation to be performed responsive to the utterance comprises changing a state of a user device comprising the first user interface;

determining, whether the operation can be performed at the user device responsive to the utterance and without requesting information from a server, wherein determining that the operation can be performed by the user device responsive to the utterance and without requesting information from the server comprises:

determining that the user device does not require data from the server to change the state responsive to the utterance, and

determining that the user device includes one or more rules associated with changing the state responsive to the utterance; and

based on determining, that the operation can be performed by the user device responsive to the utterance and without requesting information from the server:

causing, based on the one or more terms of the transcription corresponding to the utterance and one or more of the rules associated with the operation to be performed responsive to the utterance, an update to a grammar stored locally on the user device to be provided,

wherein, when it is determined, that the operation cannot be performed by the user device responsive to the utterance and without requesting information from the server, no update is provided to the grammar stored locally on the user device;

subsequent to, and in response to, causing the update to the grammar stored locally on the user device to be provided:

identifying a user profile that is associated with the user device, wherein identifying the user profile that is associated with the user device comprises identifying the user profile based on the user profile being associated with the user device and the user device being the user device to which the update to the grammar is provided;

identifying, based on the user profile, one or more other user devices that are registered to a user of the user device, wherein identifying the one or more other user devices based on the user profile is in response to the user profile being identified as associated with the user device;

determining, whether the operation can be performed by one or more of the other user devices responsive to the utterance and without requesting information from the server; and

causing, based on the identification of an other user device of the other user devices that are registered to the user of the user device, based on determining the operation can be performed by the other user device responsive to the utterance and without requesting information from the server, and based on the one or more terms, an update to an additional grammar stored locally on the other user device,

 wherein, when it is determined, that the operation cannot be performed by the other user device responsive to the utterance and without requesting information from the server, no update is provided to the additional grammar stored locally on the other user device.

9. The system of claim 8 , wherein causing the update to the grammar stored locally on the user device to be provided includes causing provision of the one or more terms to the user device.

10. The system of claim 9 , the operations further comprising:

associating the one or more terms with a predetermined time period, wherein the predetermined time period indicates a maximum period of time the one or more terms will be included in the grammar absent receiving a subsequent utterance of the one or more terms by the user to perform the operation.

11. The system of claim 8 , the operations further comprising:

causing generation, by the server, of a data object that, when executed, causes the user device to initiate performance of the operation responsive to the utterance.

12. The system of claim 11 , wherein causing an update to the grammar stored locally on the user device to be provided further comprises causing provision of the generated data object to the user device.

13. The system of claim 8 , wherein the update to the grammar enables the user device to execute a data object that initiates performance of the operation in response to subsequently receiving the utterance, and without requesting information from the server.

14. The system of claim 8 , wherein the operations are locally performed at the user device and/or the other user devices.

15. The system of claim 8 , wherein the operations are remotely performed at the server.

16. The system of claim 8 , wherein a first set of the operations are locally performed at the user device and/or the other user devices, and a second set of the operations are remotely performed at a server.

17. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

receiving a transcription of an utterance that was detected via a first user interface input, wherein the transcription of the utterance includes one or more terms corresponding to the utterance;

identifying, based on the terms of the transcription of the utterance, an operation to be performed responsive to the utterance, wherein the operation to be performed responsive to the utterance comprises changing a state of a user device comprising the first user interface;

determining, whether the operation can be performed at the user device responsive to the utterance and without requesting information from a server, wherein determining that the operation can be performed by the user device responsive to the utterance and without requesting information from the server comprises:

determining that the user device does not require data from the server to change the state responsive to the utterance, and

determining that the user device includes one or more rules associated with changing the state responsive to the utterance; and

based on determining, that the operation can be performed by the user device responsive to the utterance and without requesting information from the server:

causing, based on the one or more terms of the transcription corresponding to the utterance and one or more of the rules associated with the operation to be performed responsive to the utterance, an update to a grammar stored locally on the user device to be provided,

wherein, when it is determined, that the operation cannot be performed by the user device responsive to the utterance and without requesting information from the server, no update is provided to the grammar stored locally on the user device;

subsequent to, and in response to, causing the update to the grammar stored locally on the user device to be provided:

identifying a user profile that is associated with the user device, wherein identifying the user profile that is associated with the user device comprises identifying the user profile based on the user profile being associated with the user device and the user device being the user device to which the update to the grammar is provided;

identifying, based on the user profile, one or more other user devices that are registered to a user of the user device, wherein identifying the one or more other user devices based on the user profile is in response to the user profile being identified as associated with the user device;

determining, whether the operation can be performed by one or more of the other user devices responsive to the utterance and without requesting information from the server; and

causing, based on the identification of an other user device of the other user devices that are registered to the user of the user device, based on determining the operation can be performed by the other user device responsive to the utterance and without requesting information from the server, and based on the one or more terms, an update to an additional grammar stored locally on the other user device,

wherein, when it is determined, that the operation cannot be performed by the other user device responsive to the utterance and without requesting information from the server, no update is provided to the additional grammar stored locally on the other user device.

18. The non-transitory computer-readable medium of claim 17 , wherein causing the update to the grammar stored locally on the user device to be provided includes causing provision of the one or more terms to the user device.

19. The non-transitory computer-readable medium of claim 18 , the operations further comprising:

associating the one or more terms with a predetermined time period, wherein the predetermined time period indicates a maximum period of time the one or more terms will be included in the grammar absent receiving a subsequent utterance of the one or more terms by the user to perform the operation.

20. The non-transitory computer-readable medium of claim 17 , the operations further comprising:

causing generation, by the server, of a data object that, when executed, causes the user device to initiate performance of the operation responsive to the utterance.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 2, 2023
From: AGGARWAL, VIKRAM; GALI, MOISES MORGENSTERN
To: GOOGLE LLC
Reel/Frame 065439/0158 →
Continuity (3)
Continuation 17521472 · Nov 8, 2021
Continuation 15862615 · Jan 4, 2018
Related Publication 20240038219A1 · Feb 1, 2024
References Cited (66)
US 6195641B1 · Loring et al. · 2001 [cited by applicant]
US 6873951B1 · Lin et al. · 2005 [cited by applicant]
US 8185392B1 · Strope et al. · 2012 [cited by applicant]
US 8370146B1 · Schalkwyk et al. · 2013 [cited by applicant]
US 8805684B1 · Aleksic · 2014 [cited by examiner]
US 9070367B1 · Hoffmeister et al. · 2015 [cited by applicant]
US 9836527B2 · Gao et al. · 2017 [cited by applicant]
US 10109273B1 · Rajasekaram · 2018 [cited by examiner]
US 11170762B2 · Aggarwal et al. · 2021 [cited by applicant]
US 20020077823A1 · Fox et al. · 2002 [cited by applicant]
US 20030046074A1 · Ruback et al. · 2003 [cited by applicant]
US 20050096753A1 · Arling et al. · 2005 [cited by applicant]
US 20050105712A1 · Williams et al. · 2005 [cited by applicant]
US 20070005360A1 · Huning et al. · 2007 [cited by applicant]
US 20070073540A1 · Hirakawa et al. · 2007 [cited by applicant]
US 20070276651A1 · Bliss et al. · 2007 [cited by applicant]
US 20080235021A1 · Cross et al. · 2008 [cited by applicant]
US 20090112605A1 · Gupta · 2009 [cited by applicant]
US 20100179812A1 · Jang et al. · 2010 [cited by applicant]
US 20120179471A1 · Newman et al. · 2012 [cited by applicant]
US 20130073293A1 · Jang · 2013 [cited by examiner]
US 20130088419A1 · Kim et al. · 2013 [cited by applicant]
US 20130132089A1 · Fanty et al. · 2013 [cited by applicant]
US 20130204967A1 · Seo · 2013 [cited by examiner]
US 20130211824A1 · Tseng · 2013 [cited by examiner]
US 20140136013A1 · Wolverton et al. · 2014 [cited by applicant]
US 20140200737A1 · Lortz et al. · 2014 [cited by applicant]
US 20140244259A1 · Rosario et al. · 2014 [cited by applicant]
US 20140274203A1 · Ganong, III et al. · 2014 [cited by applicant]
US 20140337007A1 · Waibel et al. · 2014 [cited by applicant]
US 20150045988A1 · Gusikhin et al. · 2015 [cited by applicant]
US 20150255068A1 · Kim · 2015 [cited by examiner]
US 20150281401A1 · Le et al. · 2015 [cited by applicant]
US 20150371628A1 · Kreifeldt · 2015 [cited by examiner]
US 20160094509A1 · Ye · 2016 [cited by applicant]
US 20160109322A1 · Eguchi · 2016 [cited by examiner]
US 20160155443A1 · Khan et al. · 2016 [cited by applicant]
US 20160179462A1 · Bjorkengren · 2016 [cited by applicant]
US 20160252972A1 · Kim · 2016 [cited by examiner]
US 20160269524A1 · Stottlemyer · 2016 [cited by applicant]
US 20160322044A1 · Jung · 2016 [cited by examiner]
US 20160350320A1 · Sung et al. · 2016 [cited by applicant]
US 20170069312A1 · Sundararajan et al. · 2017 [cited by applicant]
US 20170162204A1 · Wilson et al. · 2017 [cited by applicant]
US 20170242914A1 · Gao · 2017 [cited by examiner]
US 20180122365A1 · Kunitake et al. · 2018 [cited by applicant]
US 20180174580A1 · Kim et al. · 2018 [cited by applicant]
US 20180217810A1 · Agrawal · 2018 [cited by applicant]
US 20190019516A1 · Lawrence et al. · 2019 [cited by applicant]
US 20190034405A1 · Ivan · 2019 [cited by examiner]
US 20190108832A1 · Tomar et al. · 2019 [cited by applicant]
US 20190206388A1 · Aggarwal et al. · 2019 [cited by applicant]
US 20200394465A1 · Gil Ramos · 2020 [cited by examiner]
US 20220157318A1 · Sharifi · 2022 [cited by examiner]
CN 101075434 · 2007 [cited by applicant]
CN 101454775 · 2009 [cited by applicant]
CN 101588417 · 2009 [cited by applicant]
CN 107093425 · 2017 [cited by applicant]
WO 2016191319 · 2016 [cited by applicant]
China National Intellectual Property Administration; Notification of Grant issued in Application No. 201980011141.4; 6pages; dated Nov. 26, 2023. [cited by applicant]
European Patent Office, Intention to Grant issued in Application No. 19701422.8, 44 pages, dated May 28, 2024. [cited by applicant]
European Patent Office; Communication pursuant Article 94(3) EPC issued in Application No. 19701422.8; 4 pages; dated Jul. 6, 2023. [cited by applicant]
China National Intellectual Property Administration; Notification of Second Office Action issued in Application No. 201980011141.4; 7 pages; dated Jul. 20, 2023. [cited by applicant]
China National Intellectual Property Administration; Notification of First Office Action issued in Application No. 201980011141.4; 30 pages; dated Dec. 5, 2022. [cited by applicant]
European Patent Office; Communication issued in Application No. 19701422.8, 4 pages, dated Oct. 20, 2022. [cited by applicant]
European Patent Office; International Search Report issued in Application No. PCT/US2019/012399; 14 pages; dated mailed Apr. 11, 2019. [cited by applicant]