IP Library › Granted Patent US 10,026,419
Granted Patent B2
US 10,026,419 · App. 14/645,802 · Granted Jul 17, 2018

Multisensory speech detection

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,026,419
App. No.
14/645,802
Granted
Jul 17, 2018
Kind
B2
Abstract

A computer-implemented method of multisensory speech detection is disclosed. The method comprises determining an orientation of a mobile device and determining an operating mode of the mobile device based on the orientation of the mobile device. The method further includes identifying speech detection parameters that specify when speech detection begins or ends based on the determined operating mode and detecting speech from a user of the mobile device based on the speech detection parameters.

Claims (51)

1. A computer-implemented method, comprising:

initiating, by a mobile computing device, an audio recording process using a microphone of the mobile computing device;

identifying, by the mobile computing device, a first orientation of the mobile computing device at a beginning of the audio recording process;

determining, during the audio recording process, that the mobile computing device has transitioned from the first orientation to a second orientation;

in response to determining that the mobile computing device has transitioned from the first orientation to a second orientation, determining a speech energy threshold for comparing to speech received during the audio recording process and while the mobile computing device is in the second orientation;

comparing an energy of the speech received during the audio recording process and while the mobile computing device is in the second orientation to the speech energy threshold;

determining an end of speech condition based on (i) the energy of the speech received during the audio recording process and while the mobile computing device is in the second orientation and (ii) the speech energy threshold; and

stopping, based on determining the end of speech condition, the audio recording process.

2. The method of claim 1 , wherein identifying the first orientation of the mobile computing device is performed in response to a proximity sensor of the mobile computing device detecting an object within a predetermined distance of the proximity sensor.

3. The method of claim 1 , wherein identifying the first orientation of the mobile computing device comprises detecting that a user is within a predetermined distance of a proximity sensor of the mobile computing device, and wherein determining that the mobile computing device has transitioned from the first orientation to the second orientation includes determining that the user is no longer within the predetermined distance of the proximity sensor.

4. The method of claim 1 , wherein determining that the mobile computing device has transitioned from the first orientation to the second orientation is performed using one or more accelerometers of the mobile computing device.

5. The method of claim 1 , comprising:

classifying the transition of the mobile computing device from the first orientation to the second orientation as one of a “to mouth” gesture, a “from mouth” gesture, a “to ear” gesture, or a “from ear” gesture,

wherein determining the speech energy threshold for comparing to speech received during the audio recording process and while the mobile computing device is in the second orientation is based on classifying the transition of the mobile computing device from the first orientation to the second orientation as one of the “to mouth” gesture, the “from mouth” gesture, the “to ear” gesture, or the “from ear” gesture.

6. The method of claim 1 , wherein determining, during the audio recording process, that the mobile computing device has transitioned from the first orientation to the second orientation comprises determining that an angle between a user and the mobile computing device is outside of a selected set of angles.

7. The method of claim 1 , wherein determining, during the audio recording process, that the mobile computing device has transitioned from the first orientation to the second orientation comprises determining that the mobile computing device is beyond a predetermined distance from a portion of a user.

8. The method of claim 1 , comprising:

while the mobile computing device is in the first orientation, comparing the energy of the speech received during the audio recording process to a different speech energy threshold;

determining that the energy of the speech received during the audio recording process and while the mobile computing device is in the first orientation does is greater than the different speech energy threshold; and

based on determining that the energy of the speech received during the audio recording process and while the mobile computing device is in the first orientation is greater than the different speech energy threshold, continuing the audio recording process.

9. A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

initiating, by a mobile computing device, an audio recording process using a microphone of the mobile computing device;

identifying, by the mobile computing device, a first orientation of the mobile computing device at a beginning of the audio recording process;

determining, during the audio recording process, that the mobile computing device has transitioned from the first orientation to a second orientation;

in response to determining that the mobile computing device has transitioned from the first orientation to a second orientation, determining a speech energy threshold for comparing to speech received during the audio recording process and while the mobile computing device is in the second orientation;

comparing an energy of the speech received during the audio recording process and while the mobile computing device is in the second orientation to the speech energy threshold;

determining an end of speech condition based on (i) the energy of the speech received during the audio recording process and while the mobile computing device is in the second orientation and (ii) the speech energy threshold; and

stopping, based on determining the end of speech condition, the audio recording process.

10. The computer storage medium of claim 9 , wherein identifying the first orientation of the mobile computing device is performed in response to a proximity sensor of the mobile computing device detecting an object within a predetermined distance of the proximity sensor.

11. The computer storage medium of claim 9 , wherein identifying the first orientation of the mobile computing device comprises detecting that a user is within a predetermined distance of a proximity sensor of the mobile computing device, and wherein determining that the mobile computing device has transitioned from the first orientation to the second orientation includes determining that the user is no longer within the predetermined distance of the proximity sensor.

12. The computer storage medium of claim 9 , wherein determining that the mobile computing device has transitioned from the first orientation to the second orientation is performed using one or more accelerometers of the mobile computing device.

13. The computer storage medium of claim 9 , comprising:

classifying the transition of the mobile computing device from the first orientation to the second orientation as one of a “to mouth” gesture, a “from mouth” gesture, a “to ear” gesture, or a “from ear” gesture,

wherein determining the speech energy threshold for comparing to speech received during the audio recording process and while the mobile computing device is in the second orientation is based on classifying the transition of the mobile computing device from the first orientation to the second orientation as one of the “to mouth” gesture, the “from mouth” gesture, the “to ear” gesture, or the “from ear” gesture.

14. The computer storage medium of claim 9 , wherein determining, during the audio recording process, that the mobile computing device has transitioned from the first orientation to the second orientation comprises determining that an angle between a user and the mobile computing device is outside of a selected set of angles.

15. The computer storage medium of claim 9 , wherein determining, during the audio recording process, that the mobile computing device has transitioned from the first orientation to the second orientation comprises determining that the mobile computing device is beyond a predetermined distance from a portion of a user.

16. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

initiating, by a mobile computing device, an audio recording process using a microphone of the mobile computing device;

identifying, by the mobile computing device, a first orientation of the mobile computing device at a beginning of the audio recording process;

determining, during the audio recording process, that the mobile computing device has transitioned from the first orientation to a second orientation;

in response to determining that the mobile computing device has transitioned from the first orientation to a second orientation, determining a speech energy threshold for comparing to speech received during the audio recording process and while the mobile computing device is in the second orientation;

comparing an energy of the speech received during the audio recording process and while the mobile computing device is in the second orientation to the speech energy threshold;

determining an end of speech condition based on (i) the energy of the speech received during the audio recording process and while the mobile computing device is in the second orientation and (ii) the speech energy threshold; and

stopping, based on determining the end of speech condition, the audio recording process.

17. The system of claim 16 , wherein identifying the first orientation of the mobile computing device is performed in response to a proximity sensor of the mobile computing device detecting an object within a predetermined distance of the proximity sensor.

18. The system of claim 16 , wherein identifying the first orientation of the mobile computing device comprises detecting that a user is within a predetermined distance of a proximity sensor of the mobile computing device, and wherein determining that the mobile computing device has transitioned from the first orientation to the second orientation includes determining that the user is no longer within the predetermined distance of the proximity sensor.

19. The system of claim 16 , wherein determining that the mobile computing device has transitioned from the first orientation to the second orientation is performed using one or more accelerometers of the mobile computing device.

20. The system of claim 16 , wherein the operations further comprise:

classifying the transition of the mobile computing device from the first orientation to the second orientation as one of a “to mouth” gesture, a “from mouth” gesture, a “to ear” gesture, or a “from ear” gesture,

wherein determining the speech energy threshold for comparing to speech received during the audio recording process and while the mobile computing device is in the second orientation is based on classifying the transition of the mobile computing device from the first orientation to the second orientation as one of the “to mouth” gesture, the “from mouth” gesture, the “to ear” gesture, or the “from ear” gesture.

Assignments (2)
CHANGE OF NAME Recorded Dec 5, 2017
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 044695/0115 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 13, 2015
From: BURKE, DAVE; LEBEAU, MICHAEL J.; GIANNO, KONRAD; KRISTJANSSON, TRAUSTI T.; JITKOFF, JOHN NICHOLAS; SENIOR, ANDREW W.
To: GOOGLE INC.
Reel/Frame 036064/0670 →