IP Library Granted Patent US 11,289,086
Granted Patent B2
US 11,289,086 · App. 16/724,096 · Granted Mar 29, 2022

Selective response rendering for virtual assistants

Inventors: Nicholas David Burton (Bellevue, WA); Arash Ghanaie-Sichanie (Woodinville, WA); Qi Liu (Bellevue, WA); Senthil Kumar Velayutham (Sammamish, WA); Jian Wu (Bellevue, WA)
Assignee: Microsoft Technology Licensing, LLC
G10L15/22G06F3/012G06F3/013G06F3/167G06F9/453G10L15/08G10L15/25G10L25/51G10L2015/088G10L2015/223
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,289,086
App. No.
16/724,096
Granted
Mar 29, 2022
Kind
B2
Abstract

A system and method for selecting a target device out of a larger group of candidate devices for rendering a response from a virtual assistant to an end-user is disclosed. The system determines that a same trigger phrase included in an utterance has been received by multiple devices that are in proximity to one another at around the same time. These candidate devices can collect attention data, such as user gaze toward a device, to select the device that was most likely the intended recipient of the utterance. The system is configured to control the virtual assistant to render a response solely via the selected device.

Claims (68)

1. A system for selecting one of a first device and a second device to respond to a voice command, wherein the second device is in proximity to the first device, the system comprising:

one or more processors; and

one or more machine-readable media including instructions stored therein which, when executed by the one or more processors, cause the system to:

determine a first spoken utterance captured as a first audio signal by the first device around a first time including a first trigger phrase for engaging with a virtual assistant;

determine the first spoken utterance captured as a second audio signal by the second device around the first time;

receive, from the first device, a first attention signal captured by the first device around the first time;

receive from the second device, a second attention signal captured by the second device around the first time;

determine, based on a comparison of the first attention signal and the second attention signal, that the first device was the most likely intended recipient device of the first trigger phrase among the first device and the second device;

render a response to the first spoken utterance via the first device in response to the determination that the first device was the most likely intended recipient device; and

cause the second device to not render a response to the first spoken utterance in response to the determination that the first device was the most likely intended recipient device of the first trigger phrase.

2. The system of claim 1 , wherein the first attention signal includes eye tracking data for a first user, and the eye tracking data indicates the first user was gazing at the first device around the first time.

3. The system of claim 1 , wherein the first attention signal includes head orientation data for a first user, and the head orientation data indicates the first user was facing toward the first device around the first time.

4. The system of claim 1 , wherein the first attention signal includes lip movement or lip-reading data for a first user, and the lip movement or lip-reading data indicates the first user articulated the spoken utterance at the first time.

5. The system of claim 1 , wherein the first audio signal is determined to have a greater amplitude than the second audio signal.

6. The system of claim 1 , wherein the first audio signal is determined to have less noise than the second audio signal.

7. The system of claim 1 , wherein the first device is associated with a first user account for a first user and the second device is associated with a second user account for a second user, and the instructions further cause the system to:

access a first calendar of the first user account and a second calendar of the second user account; and

ascertain, based on the first calendar and the second calendar, that the first user is scheduled to be near the first device at the first time and the second user is scheduled to be away from the second device at the first time,

wherein the determining that the second device is less likely to represent the intended recipient device than the first device is further based on ascertaining that the first user is scheduled to be near the first device and the second user is scheduled to be away from the second device.

8. The system of claim 1 , wherein the instructions further cause the system to:

determine a second spoken utterance, captured as a third audio signal by the first device around a second time and as a fourth audio signal by the second device around the second time, includes a second trigger phrase;

receive a third attention signal captured by the first device around the second time and a fourth attention signal captured by the second device around the second time;

determine, based on the third attention signal and the fourth attention signal, that the second device was the most likely intended recipient device of the second trigger phrase; and

render a response to the second spoken utterance via only the second device in response to the determination that the second device was the most likely intended recipient device of the second trigger phrase.

9. The system of claim 1 , wherein the first attention signal is obtained from an optical proximity device included in the first device.

10. A method of selecting one of a first device and a second device to respond to a voice command, wherein the second device is in proximity to the first device, to provide access to a virtual assistant, the method comprising:

determining a first spoken utterance captured as a first audio signal by the first device around a first time including a first trigger phrase for engaging with the virtual assistant;

determining the first spoken utterance captured as a second audio signal by the second device around the first time;

receiving, from the first device, a first attention signal captured by the first device around the first time;

receiving from the second device, a second attention signal captured by the second device around the first time;

determining, based on a comparison of the first attention signal and the second attention signal, that the first device was the most likely intended recipient device of the first trigger phrase among the first device and the second device;

rendering a response to the first spoken utterance via the first device in response to the determination that the first device was the most likely intended recipient device; and

causing the second device to not render a response to the first spoken utterance in response to the determination that the first device was the most likely intended recipient device of the first trigger phrase.

11. The method of claim 10 , wherein the first attention signal includes eye tracking data for a first user, and the eye tracking data indicates the first user was gazing at the first device around the first time.

12. The method of claim 10 , wherein the first attention signal includes head orientation data for a first user, and the head orientation data indicates the first user was facing toward the first device around the first time.

13. The method of claim 10 , wherein the first attention signal includes lip-reading data for a first user, and the lip-reading data indicates the first user articulated the spoken utterance at the first time.

14. The method of claim 10 , wherein the first audio signal has a greater amplitude than the second audio signal.

15. The method of claim 10 , wherein the first audio signal has less noise than the second audio signal.

16. The method of claim 10 , wherein the first device is associated with a first user account for a first user and the second device is associated with a second user account for a second user, and the method further comprises:

accessing a first calendar of the first user account and a second calendar of the second user account; and

ascertaining, based on the first calendar and the second calendar, that the first user is scheduled to be near the first device at the first time and the second user is scheduled to be away from the second device at the first time,

wherein the determining that the second device is less likely to represent the intended recipient device than the first device is further based on ascertaining that the first user is scheduled to be near the first device and the second user is scheduled to be away from the second device.

17. The method of claim 10 , further comprising:

determining a second spoken utterance, captured as a third audio signal by the first device around a second time and as a fourth audio signal by the second device around the second time, includes a second trigger phrase;

receiving a third attention signal captured by the first device around the second time and a fourth attention signal captured by the second device around the second time;

determining, based on the third attention signal and the fourth attention signal, that the second device was the most likely intended recipient device of the second trigger phrase; and

rendering a response to the second spoken utterance via only the second device in response to the determination that the second device was the most likely intended recipient device of the second trigger phrase.

18. The method of claim 10 , wherein the first device includes an optical proximity device.

19. The system of claim 1 , wherein:

the first spoken utterance is made by a first user within listening ranges of both the first device and the second device;

the first attention signal is determined from attention data regarding head orientation of the first user obtained by a first camera on the first device;

the second attention signal is determined from attention data regarding head orientation of the first user obtained by a second camera on the second device, and

the first and second devices are both virtual assistant-enabled devices and are both connected to a selective virtual assistant response management system configured to select which of the first and second devices will be selected to render the response to the first spoken utterance based on the first and second attention signals.

20. The system of claim 1 , wherein:

the first spoken utterance is made by a first user within listening ranges of both the first device and the second device;

the first attention signal is determined from attention data regarding eye-gaze tracking of the first user obtained by a first camera on the first device;

the second attention signal is determined from attention data regarding eye-gaze tracking of the first user obtained by a second camera on the second device, and

the first and second devices are both virtual assistant-enabled devices and are both connected to a selective virtual assistant response management system configured to select which of the first and second devices will be selected to render the response to the first spoken utterance based on the first and second attention signals.

21. The method of claim 10 , wherein:

the first spoken utterance is made by a first user within listening ranges of both the first device and the second device;

the first attention signal is determined from attention data regarding head orientation of the first user obtained by a first camera on the first device;

the second attention signal is determined from attention data regarding head orientation of the first user obtained by a second camera on the second device, and

the first and second devices are both virtual assistant-enabled devices and are both connected to a selective virtual assistant response management system configured to select which of the first and second devices will be selected to render the response to the first spoken utterance based on the first and second attention signals.

22. The method of claim 10 , wherein:

the first spoken utterance is made by a first user within listening ranges of both the first device and the second device;

the first attention signal is determined from attention data regarding eye-gaze tracking of the first user obtained by a first camera on the first device;

the second attention signal is determined from attention data regarding eye-gaze tracking of the first user obtained by a second camera on the second device, and

the first and second devices are both virtual assistant-enabled devices and are both connected to a selective virtual assistant response management system configured to select which of the first and second devices will be selected to render the response to the first spoken utterance based on the first and second attention signals.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 3, 2020
From: BURTON, NICHOLAS DAVID; GHANAIE-SICHANIE, ARASH; LIU, QI; VELAYUTHAM, SENTHIL KUMAR; WU, JIAN
To: MICROSOFT TECHNOLOGY LICENSING, LLC
Reel/Frame 052310/0658 →
Continuity (2)
Provisional Application 62929731 · Nov 1, 2019
Related Publication 20210134286A1 · May 6, 2021
Cited By (2)
US 12,334,067 US 12,406,671