IP Library Granted Patent US 10,438,587
Granted Patent B1
US 10,438,587 · App. 15/671,613 · Granted Oct 8, 2019

Speech recognition biasing

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,438,587
App. No.
15/671,613
Granted
Oct 8, 2019
Kind
B1
Abstract

Systems and methods are described include a robot and/or an associated computing system that can use various cues about an environment of the robot to apply a bias to increase the accuracy of speech transcription. In some implementations, audio data corresponding to a spoken instruction to a robot is received. Candidate transcriptions of the audio data are obtained. A respective action of the robot corresponding to each of the candidate transcriptions of the audio data is determined. One or more scores indicating characteristics of a potential outcome of performing the respective action corresponding to the candidate transcription of the audio data are determined for each of the candidate transcriptions of the audio data. A particular candidate transcription is selected from among the candidate transcriptions based at least on the one or more scores. The action determined for the particular candidate transcription is performed.

Claims (58)

1. A method comprising:

receiving audio data corresponding to a spoken instruction to a robot;

receiving context data that indicates (i) a location of the robot, and (ii) one or more objects within a threshold level of proximity to the location of the robot;

obtaining candidate transcriptions of the audio data;

determining a respective action of the robot corresponding to each of the candidate transcriptions of the audio data;

determining, for each of the candidate transcriptions of the audio data, one or more scores indicating characteristics of a potential outcome of performing the respective action corresponding to the candidate transcription of the audio data, wherein the one or more scores are determined based at least on the received context data;

selecting a particular candidate transcription from among the candidate transcriptions based at least on the one or more scores; and

performing the action determined for the particular candidate transcription.

2. The method of claim 1 , wherein the context data is collected by one or more on-board sensors of the robot.

3. The method of claim 1 , wherein the one or more scores each represent a respective likelihood that performing the respective action will result in injury to a user that provides the spoken instruction to the robot.

4. The method of claim 1 , further comprising:

determining, for each of the candidate transcriptions of the audio data, a confidence score that reflects a likelihood that the audio data will be transcribed using the candidate transcription; and

biasing the confidence scores for the candidate transcription scores based at least on the one or more scores.

5. The method of claim 4 , wherein biasing the confidence scores for the candidate transcription scores comprises:

determining that the potential outcome of performing a particular action corresponding to a particular candidate transcription of the audio data is included within a set of undesirable outcomes that is associated with the robot; and

reducing a confidence score for the particular candidate transcription of the audio data based on determining that the potential outcome of performing the particular action is included within the set of undesirable outcomes that is associated with the robot.

6. The method of claim 1 , further comprising:

computing a recognition score for each of the candidate transcriptions;

computing an impact score for each of the candidate transcriptions; and

combining, for each of the candidate transcriptions, the recognition score and the impact score to compute a confidence score that reflects a likelihood that the audio data will be transcribed using the candidate transcription.

7. The method of claim 6 , wherein, for each candidate transcription:

the recognition score represents a likelihood that the candidate transcription represents an accurate transcription of the spoken instruction provided to the robot; and

the impact score represents a likelihood that the potential outcome of performing a respective action corresponding to the candidate transcription is included among a set of undesirable outcomes that is associated with the robot.

8. A system comprising:

one or more computers; and

one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

receiving audio data corresponding to a spoken instruction to a robot;

receiving context data that indicates (i) a location of the robot, and (ii) one or more objects within a threshold level of proximity to the location of the robot;

obtaining candidate transcriptions of the audio data;

determining a respective action of the robot corresponding to each of the candidate transcriptions of the audio data;

determining, for each of the candidate transcriptions of the audio data, one or more scores indicating characteristics of a potential outcome of performing the respective action corresponding to the candidate transcription of the audio data, wherein the one or more scores are determined based at least on the received context data;

selecting a particular candidate transcription from among the candidate transcriptions based at least on the one or more scores; and

performing the action determined for the particular candidate transcription.

9. The system of claim 8 , wherein the context data is collected by one or more on-board sensors of the robot.

10. The system of claim 8 , wherein the one or more scores each represent a respective likelihood that performing the respective action will cause injury to a user that provides the spoken instruction to the robot.

11. The system of claim 8 , wherein the operations further comprise:

determining, for each of the candidate transcriptions of the audio data, a confidence score that reflects a likelihood that the audio data will be transcribed using the candidate transcription; and

biasing the confidence scores for the candidate transcription scores based at least on the one or more scores.

12. The system of claim 11 , wherein biasing the confidence scores for the candidate transcription scores comprises:

determining that the potential outcome of performing a particular action corresponding to a particular candidate transcription of the audio data is included within a set of undesirable outcomes that is associated with the robot; and

reducing a confidence score for the particular candidate transcription of the audio data based on determining that the potential outcome of performing the particular action is included within the set of undesirable outcomes that is associated with the robot.

13. The system of claim 8 , further comprising:

computing a recognition score for each of the candidate transcriptions;

computing an impact score for each of the candidate transcriptions; and

combining, for each of the candidate transcriptions, the recognition score and the impact score to compute a confidence score that reflects a likelihood that the audio data will be transcribed using the candidate transcription.

14. A non-transitory computer-readable storage device encoded with computer program instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

receiving audio data corresponding to a spoken instruction to a robot;

receiving context data that indicates (i) a location of the robot, and (ii) one or more objects within a threshold level of proximity to the location of the robot;

obtaining candidate transcriptions of the audio data;

determining a respective action of the robot corresponding to each of the candidate transcriptions of the audio data;

determining, for each of the candidate transcriptions of the audio data, one or more scores indicating characteristics of a potential outcome of performing the respective action corresponding to the candidate transcription of the audio data, wherein the one or more scores are determined based at least on the received context data;

selecting a particular candidate transcription from among the candidate transcriptions based at least on the one or more scores; and

performing the action determined for the particular candidate transcription.

15. The non-transitory computer-readable storage device of claim 14 , wherein the context data is collected by one or more on-board sensors of the robot.

16. The non-transitory computer-readable storage device of claim 14 , wherein the one or more scores each represent a respective likelihood that performing the respective action will cause injury to a user that provides the spoken instruction to the robot.

17. The non-transitory computer-readable storage device of claim 14 , wherein the operations further comprise:

determining, for each of the candidate transcriptions of the audio data, a confidence score that reflects a likelihood that the audio data will be transcribed using the candidate transcription; and

biasing the confidence scores for the candidate transcription scores based at least on the one or more scores.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 29, 2025
From: GOOGLE LLC
To: GDM HOLDING LLC
Reel/Frame 071109/0342 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 21, 2023
From: X DEVELOPMENT LLC
To: GOOGLE LLC
Reel/Frame 064658/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 14, 2017
From: LAM, DANIEL ALEX
To: X DEVELOPMENT LLC
Reel/Frame 043280/0102 →