Speech recognition using multiple sensors
Introduced here are systems and methods to increase accuracy of speech recognition by utilizing multiple sensors placed at multiple speech transmission regions such as a user's lips, throat, ear canal, etc. The various speech transmission regions are better at transmitting certain phonemes, and a sensor placed close to a particular speech transition region can more accurately detect the phoneme transmitted through the particular speech transmission region. For example, a microphone placed close to the lips can better detect labial phonemes such as m, n, p, and b, than a microphone placed close to the throat. Further, disclosed here are ways to reduce energy consumption while performing speech recognition.
1. A system comprising:
an earbud comprising a first microphone, to be disposed at an entrance or within a user's ear canal, to measure a first sound transmitted through the user's ear canal;
a sensor comprising a second microphone, to be disposed proximate to a user's lips, to measure a second sound transmitted through the user's lips, wherein said sensor is physically separate from the earbud and wherein said sensor is configured to be secured below a head and on a body of the user; and
a processor configured to:
receive a first measurement of the first sound and a second measurement of the second sound;
enhance an accuracy of a speech recognition algorithm by determining a difference between a part of the first sound and a part of the second sound, the difference between the part of the first sound and the part of the second sound comprising a difference between an amplitude and a phase associated with the first sound and the second sound recorded from each microphone;
modify probabilities of phoneme predictions based on the difference; and
selectively focus on one of the first or the second microphones based on the probabilities of the phoneme predictions.
2. The system of claim 1 , comprising the processor configured to modify probabilities of phoneme predictions comprising the processor configured to:
reconstruct a user's speech by selecting a labial phoneme from the second microphone and a non-labial phoneme from the first microphone.
3. The system of claim 1 , comprising:
a third microphone, to be disposed proximate to a user's throat, to measure a third sound transmitted through the user's throat.
4. The system of claim 3 , comprising the processor configured to:
reconstruct a user's speech by selecting a laryngeal phoneme from the third microphone and a non-labial and non-laryngeal phoneme from the first microphone.
5. The system of claim 1 , comprising the processor configured to:
recognize an activating vocalization based on the first measurement and the second measurement; and
upon recognizing the activating vocalization, facilitate a recognition of a user's speech.
6. The system of claim 1 , comprising the processor configured to:
reduce energy consumption associated with the second microphone by operating the second microphone in a low-energy mode until the first microphone detects the first sound, and upon detecting the first sound, transitioning the second microphone into a high-energy mode to measure the second sound.
7. A system comprising:
a plurality of sensors, a first sensor in the plurality of sensors to be disposed at an entrance of or within a user's ear canal and a second sensor in the plurality of sensors to be disposed in proximity to a speech transmission region associated with a user's speech system, the first sensor to sense a first sound in the user's ear canal and the second sensor to sense a second sound transmitted by the speech transmission region, wherein said second sensor is physically separate from the second sensor and wherein said second sensor is configured to be secured below a head and on a body of the user; and
a processor configured to:
receive one or more measurements of the first sound and the second sound;
enhance an accuracy of a speech recognition algorithm by determining a difference between a part of the first sound and a part of the second sound and based on a difference modify probabilities of phoneme predictions; and
determine a least one sensor in the plurality of sensors likely to sense a phoneme based on the probabilities of phoneme predictions and the one or more measurements of the first sound and the second sound.
8. The system of claim 7 , comprising the processor configured to enhance the accuracy of the speech recognition algorithm comprising the processor configured to:
determine that the part of the first sound has a higher amplitude than the part of the second sound; and
rely more on the part of the first sound than the part of the second sound in the speech recognition algorithm.
9. The system of claim 7 , comprising the processor configured to:
reconstruct a user's speech by selecting a first part of speech from the first sound and a second part of speech from the second sound based on a criterion indicating which sensor among the first and the second sensor is better at sensing the first part of speech and the second part of speech.
10. The system of claim 9 , the processor configured to reconstruct the user's speech comprising the processor configured to:
select a phoneme from a sound recorded by a sensor placed closer to a transmission source of the phoneme than at least one other sensor.
11. The system of claim 10 , the processor configured to select the phoneme comprising the processor configured to:
select labial phonemes from the second sound when the second sensor is placed closer to a user's lips than the first sensor.
12. The system of claim 9 , the processor configured to reconstruct the user's speech comprising the processor configured to:
obtain the criterion indicating a frequency range; and
select a low frequency sound from the first sensor and a high frequency sound from the second sensor.
13. The system of claim 7 , comprising the processor configured to:
recognize an activating vocalization based on the one or more measurements; and
upon recognizing the activating vocalization, facilitate recognition of a user's speech.
14. The system of claim 13 , comprising the processor to: continuously operate the at least one sensor in a high-energy mode.
15. The system of claim 14 , comprising the processor configured to recognize the activating vocalization when the sensor senses the one or more phonemes.
16. The system of claim 13 , the processor comprising a dual-mode processor configured to operate in a low-energy mode prior to recognizing the activating vocalization and to operate in a high-energy mode when facilitating the recognition of the user's speech.
17. The system of claim 7 , comprising one or more sensors in the plurality of sensors to be disposed outside of the user's head and exposed to an ambient sound, the one or more sensors to extract a user's speech by removing the ambient sound from a sensed audio.
18. The system of claim 7 , comprising:
an enclosure surrounding a sensor in the plurality of sensors, the enclosure to attenuate ambient sound reaching the sensor; and
the processor to reduce energy consumption by:
receiving a notification from the sensor surrounded by the enclosure, the notification indicating a detection of a user's speech, and
upon receiving the notification activating a remainder of sensors in the plurality of sensors to sense the user's speech.
19. The system of claim 7 , a sensor in the plurality of sensors to be disposed in proximity to user's lips, or the sensor disposed in proximity to a user's throat.
20. The system of claim 7 , comprising the processor to:
measure an otoacoustic emission generated within the user's ear canal in response to a received signal and identify the user based on the otoacoustic emission.
21. The system of claim 7 , comprising:
the processor to reduce energy consumption by receiving a user's speech from the plurality of sensors and sending the user's speech to a remote processor to perform speech recognition.
22. A method comprising:
measuring a plurality of sounds transmitted by a plurality of speech transmission regions associated with a user's speech system by a plurality of sensors, a first sensor in the plurality of sensors to be disposed at an entrance of or within a user's ear canal and a second sensor in the plurality of sensors to be disposed in proximity to a speech transmission region, the first sensor to measure a first sound at the entrance of or within the user's ear canal and the second sensor to measure a second sound transmitted by the speech transmission region, wherein said second sensor is physically separate from the second sensor and wherein said second sensor is configured to be secured below a head and on a body of the user;
enhancing an accuracy of a speech recognition algorithm by determining a difference between a part of the first sound and a part of the second sound and based on a difference modify probabilities of phoneme predictions; and
determining a least one sensor in the plurality of sensors likely to sense a phoneme based on the probabilities of phoneme predictions, the first sound, and the second sound.
23. The method of claim 22 , comprising:
reconstructing a user's speech by selecting a first part of speech from the first sound and a second part of speech from the second sound based on a criterion indicating which sensor among the first and the second sensor is better at sensing the first part of speech and the second part of speech.
24. The method of claim 23 , said selecting comprising:
selecting a phoneme from a sound recorded by a sensor placed closer to a transmission source of the phoneme than at least one other sensor.
25. The method of claim 24 , said selecting comprising:
selecting labial phonemes from the second sound when the second sensor is placed closer to a user's lips than the first sensor.
26. The method of claim 23 , said selecting comprising:
selecting a low frequency sound from the first sensor and a high frequency sound from the second sensor.
27. The method of claim 22 , comprising:
recognizing an activating vocalization based on a plurality of measurements of the first sound and the second sound; and
upon recognizing the activating vocalization, facilitating recognition of a user's speech.
28. The method of claim 27 , comprising:
based on one or more phonemes contained in the activating vocalization, determining a sensor in the plurality of sensors likely to sense the one or more phonemes; and
continuously operating the sensor in a high-energy mode.
29. The method of claim 28 , comprising:
reducing energy consumption by transitioning a processor into the high-energy mode upon recognizing the activating vocalization, the processor configured to operate in a low-energy mode when not activated and to operate in the high-energy mode upon being activated; and
recognizing the user's speech when operating in the high-energy mode.
30. The method of claim 22 , comprising:
determining a time delay to receive a third sound at each sensor in the plurality of sensors;
identifying a source of the third sound based on the plurality of time delays; and
extracting a portion of the third sound belonging to a desired source from the third sound.
31. The method of claim 22 , comprising:
reducing energy consumption by receiving a notification from a sensor surrounded by an enclosure attenuating ambient sound reaching the sensor, the notification indicating a detection of a user's speech; and
upon receiving the notification activating a remainder of sensors in the plurality of sensors to sense the user's speech.
32. The method of claim 22 , comprising:
measuring an otoacoustic emission generated within the user's ear canal in response to a received signal and identify the user based on the otoacoustic emission.