IP Library Granted Patent US 11,558,505
Granted Patent B2
US 11,558,505 · App. 17/122,805 · Granted Jan 17, 2023

Computational assistant extension device

Inventors: Adam Bradley Golden (Mountain View, CA); William Gerard Donnelly (Sunnyvale, CA); Yonatan Zunger (Mountain View, CA); Leonardo Kusumo (San Jose, CA)
Assignee: GOOGLE LLC
H04M3/4936G06F3/167G06F16/435G06F16/90332G10L15/22H04M3/4938H04M2201/40H04M2250/74
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,558,505
App. No.
17/122,805
Granted
Jan 17, 2023
Kind
B2
Abstract

An example method includes receiving, by a computational assistant executing at one or more processors of a mobile computing device and via a wireless link between the mobile computing device and an external device, a representation of audio data generated by a microphone of the external device, the audio data representing a spoken utterance detected by the external device; determining, by the computational assistant and based on the audio data, a response to the spoken utterance; and sending, by the mobile computing device, to the external device, and via the wireless link between the mobile computing device and the external device, a command to output, for playback by one or more speakers connected to the external device via a hardwired analog removable connector of the external device or a wireless link between the external device and the one or more speakers, audio data representing the response to the spoken utterance.

Claims (54)

1. A method comprising:

determining, by an assistant extension device, that an activation action has occurred, wherein the assistant extension device does not have an Internet connection and is not able to communicate with a digital assistant system;

generating, by one or more microphones of the assistant extension device, audio data capturing a spoken utterance detected by the assistant extension device after the activation action occurred;

transmitting, by the assistant extension device and via a wireless Bluetooth link between the assistant extension device and an external smartphone, a representation of the audio data to the external smartphone, wherein transmitting the representation of the audio data is in response to the audio data being captured after the activation action occurred, and wherein the external smartphone has an Internet connection and is able to communicate, via the Internet connection, with the digital assistant system;

receiving, by the assistant extension device and via the wireless Bluetooth link between the assistant extension device and the external smartphone, a command from the external smartphone that causes the assistant extension device to output, for playback by one or more external speakers connected to the assistant extension device, and via a hardwired removable connector of the assistant extension device or a wireless Bluetooth link between the assistant extension device and the one or more external speakers, additional audio data representing a response to the spoken utterance; and

in response to receiving the command from the external smartphone

outputting, by the assistant extension device and via the hardwired removable connector or the wireless Bluetooth link between the assistant extension device and the one or more external speakers, the additional audio data representing the response to the spoken utterance.

2. The method of claim 1 , wherein the one or more microphones of the assistant extension device comprise at least one far-field microphone.

3. The method of claim 2 , wherein determining, by the assistant extension device, that the activation action has occurred, comprises:

generating, by the one or more microphones of the assistant extension device, preceding audio data detected by the assistant extension device before the spoken utterance;

determining whether the preceding audio data includes a trigger phrase; and

determining that the activation action has occurred in response to determining that the preceding audio data includes the trigger phrase.

4. The method of claim 1 , wherein determining, by the assistant extension device, that the activation action has occurred, comprises:

determining that a speech reception button of the assistant extension device has been activated.

5. The method of claim 1 , wherein the one or more external speakers connected to the assistant extension device include both a first external speaker connected via the hardwired removable connector and one or more second external speakers connected via the wireless Bluetooth link.

6. The method of claim 5 , wherein the hardwired removable connector comprises a 3.5 millimeter audio jack.

7. An assistant extension device comprising:

one or more communication units configured to establish a wireless Bluetooth link between the assistant extension device and an external smartphone at which a computational assistant is executing,

wherein the assistant extension device does not have an Internet connection and is not able to communicate with a digital assistant system, and

wherein the external smartphone has an Internet connection and is able to communicate, via the Internet connection, with the digital assistant system;

one or more microphones, including at least one far-field microphone;

a hardwired removable audio connector; and

one or more processors configured to:

determine that an activation action has occurred;

generate, by the one or more microphones, audio data capturing a spoken utterance detected by the assistant extension device after the activation action occurred;

transmit, via the wireless Bluetooth link between the assistant extension device and the external smartphone, a representation of the audio data to the external smartphone, wherein transmitting the representation of the audio data is in response to the audio data being captured after the activation action occurred;

receive, via the wireless Bluetooth link between the assistant extension device and the external smartphone, a command from the external smartphone that causes the assistant extension device to output, for playback by one or more external speakers connected to the assistant extension device via the hardwired removable audio connector of the assistant extension device or a wireless Bluetooth link between the assistant extension device and the one or more external speakers, additional audio data representing a response to the spoken utterance; and

in response to receiving the command from the external smartphone:

output, via the hardwired removable connector or the wireless Bluetooth link between the assistant extension device and the one or more external speakers, the additional audio data representing the response to the spoken utterance.

8. The assistant extension device of claim 7 , wherein the one or more microphones of the assistant extension device further include at least a second far-field microphone.

9. The assistant extension device of claim 8 , wherein, in determining that the activation action has occurred, one or more of the processors are configured to:

generate, by the one or more microphones, preceding audio data detected by the assistant extension device before the spoken utterance;

determine whether the preceding audio data includes a trigger phrase; and

determine that the activation action has occurred in response to determining that the preceding audio data includes the trigger phrase.

10. The assistant extension device of claim 7 , wherein, in determining that the activation action has occurred, one or more of the processors are configured to:

determine that a speech reception button of the assistant extension device has been activated.

11. The assistant extension device of claim 7 , wherein the one or more external speakers connected to the assistant extension device include both a first external speaker connected via the hardwired removable connector and one or more second external speakers connected via the wireless Bluetooth link.

12. The assistant extension device of claim 11 , wherein the hardwired removable connector comprises a 3.5 millimeter audio jack.

13. A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors of an assistant extension device to:

determine that an activation action has occurred, wherein the assistant extension device does not have an Internet connection and is not able to communicate with a digital assistant system;

generate, by one or more microphones of the assistant extension device, audio data capturing a spoken utterance detected by the assistant extension device after the activation action occurred;

transmit, via a wireless Bluetooth link between the assistant extension device and an external smartphone, a representation of the audio data to the external smartphone, wherein transmitting the representation of the audio data is in response to the audio data being captured after the activation action occurred, and wherein the external smartphone has an Internet connection and is able to communicate, via the Internet connection, with the digital assistant system;

receive, via the wireless Bluetooth link between the assistant extension device and the external smartphone, a command from the external smartphone that causes the assistant extension device to output, for playback by one or more external speakers connected to the assistant extension device, and via a hardwired removable connector of the assistant extension device or a wireless Bluetooth link between the assistant extension device and the one or more external speakers, additional audio data representing a response to the spoken utterance; and

in response to receiving the command from the external smartphone

output, via the hardwired removable connector or the wireless Bluetooth link between the assistant extension device and the one or more external speakers, the additional audio data representing the response to the spoken utterance.

14. The non-transitory computer-readable storage medium of claim 13 , wherein the one or more microphones of the assistant extension device comprise at least one far-field microphone.

15. The non-transitory computer-readable storage medium of claim 14 , wherein, in determining that the activation action has occurred, one or more of the processors are configured to:

generate, by the one or more microphones, preceding audio data detected by the assistant extension device before the spoken utterance;

determine whether the preceding audio data includes a trigger phrase; and

determine that the activation action has occurred in response to determining that the preceding audio data includes the trigger phrase.

16. The non-transitory computer-readable storage medium of claim 13 , wherein, in determining that the activation action has occurred, one or more of the processors are configured to:

determine that a speech reception button of the assistant extension device has been activated.

17. The non-transitory computer-readable storage medium of claim 13 , wherein the one or more external speakers connected to the assistant extension device include both a first external speaker connected via the hardwired removable connector and one or more second external speakers connected via the wireless Bluetooth link.

18. The non-transitory computer-readable storage medium of claim 17 , wherein the hardwired removable connector comprises a 3.5 millimeter audio jack.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 28, 2021
From: GOLDEN, ADAM BRADLEY; DONNELLY, WILLIAM GERARD; ZUNGER, YONATAN; KUSUMO, LEONARDO
To: GOOGLE INC.
Reel/Frame 055065/0498 →
CHANGE OF NAME Recorded Jan 28, 2021
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 055160/0599 →
Continuity (2)
Continuation 15650398 · Jul 14, 2017
Related Publication 20210105359A1 · Apr 8, 2021