Methods and apparatus for reducing latency in speech recognition applications

1. A computing device including a speech-enabled application installed thereon, the computing device comprising:

an input interface, which receives audio comprising speech from a user of the computing device;

an automatic speech recognition (ASR) engine, which:

detects an end of speech in a first audio portion of the received audio, and

generates a first ASR result based, at least in part, on the first audio portion; and

at least one processor programmed to:

determine whether a valid action can be performed by the speech-enabled application using the first ASR result; and

instruct the ASR engine to process a second audio portion of the received audio, recorded after the detected end of speech of the first audio portion, when it is determined that a valid action cannot be performed by the speech-enabled application using the first ASR result,

wherein the ASR engine processes the second audio portion in addition to the first audio portion when instructed by the at least one processor.

2. The computing device of claim 1 , wherein the at least one processor determines whether a valid action can be performed by the speech-enabled application using the first ASR result based, at least in part, on a natural language understanding (NLU) result generated using the first ASR result.

3. The computing device of claim 2 ,

wherein the at least one processor is further programmed to submit the NLU result to the speech-enabled application, and

wherein the at least one processor determines whether a valid action can be performed by the speech-enabled application using the first ASR result after receiving an indication from the speech-enabled application that a valid action cannot be performed in response to the NLU result submitted to the speech-enabled application.

4. The computing device of claim 1 , wherein the ASR engine processes the second audio portion by:

determining whether the second audio portion includes speech, and

generating a second ASR result based, at least in part, on at least a portion of the second audio portion in response to a determination that the second audio portion includes speech.

5. The computing device of claim 4 , wherein the generating of the second ASR result is based, at least in part, on at least a portion of the first audio portion and at least a portion of the second audio portion.

6. The computing device of claim 4 , wherein the at least one processor is further programmed to:

determine whether a valid action can be performed by the speech-enabled application using a natural language understanding (NLU) result generated based, at least in part, on at least a portion of the first ASR result and at least a portion of the second ASR result, and

instruct the speech-enabled application to perform the valid action in response to a determination that the valid action can be performed using the NLU result.

7. The computing device of claim 4 , further comprising:

at least one memory storing one or more prefixes, each of the one or more prefixes being associated with a corresponding threshold time useable by the ASR engine for endpointing,

wherein the at least one processor determines whether a valid action can be performed by the speech-enabled application by determining whether the speech in the first audio portion includes a prefix of the one or more prefixes stored on the at least one memory.

8. The computing device of claim 7 ,

wherein the ASR engine processes a plurality of time segments of the first audio portion prior to detecting the end of speech in the first audio portion, and

wherein the determining of whether the speech in the first audio portion includes a prefix stored on the at least one memory comprises comparing an output of the ASR engine determined based on the plurality of time segments processed by the ASR engine, to the one or more prefixes stored in the at least one memory.

9. The computing device of claim 7 , wherein the at least one processor is further programmed to:

update a threshold time useable by the ASR engine for endpointing in response to determining that the speech in the first audio portion includes a prefix stored in the at least one memory,

wherein the threshold time is updated by instructing the ASR engine to use the threshold time associated with the prefix stored in the at least one memory identified in the speech in the first audio portion to detect an end of speech in the first audio portion.

10. The computing device of claim 1 , wherein the at least one processor is further programmed to:

create a first hint based, at least in part, on the first ASR result, wherein the first hint prompts the user for speech input corresponding to a valid action that can be performed by the speech-enabled application, and

present the first hint via the input interface of the computing device.

11. The computing device of claim 10 , wherein the ASR engine processes the second audio portion to generate a second ASR result, and wherein the at least one processor is further programmed to:

create a second hint based, at least in part, on the first ASR result, or the second ASR result, or the first and second ASR results, wherein the second hint prompts the user for speech input corresponding to a valid action that can be performed by the speech-enabled application, and

present the second hint via a user interface of the computing device.

12. The computing device of claim 10 , wherein the at least one processor presents the first hint by visually displaying the first hint on the user interface, and wherein the first hint hints of additional information to supplement the first audio portion to perform the valid action.

13. The computing device of claim 10 , wherein the ASR engine performs ASR processing on the second audio portion based, at least in part, on information included in the first hint.

14. The computing device of claim 1 , further comprising:

at least one memory storing at least one data structure including information describing a plurality of natural language understanding (NLU) results and corresponding ASR outputs used to generate the plurality of NLU results;

wherein the at least one processor is further programmed to:

determine whether to add the first ASR result and a corresponding NLU result generated using the first ASR result to the at least one data structure stored in the at least one memory; and

add the first ASR result and the corresponding NLU result generated using the first ASR result to the at least one data structure stored in the at least one memory in response to a determination that the first ASR result and the corresponding NLU result should be added.

15. The computing device of claim 14 , wherein the at least one processor determines whether to add the first ASR result and the corresponding NLU result generated using the first ASR result to the at least one data structure by:

determining a number of times the corresponding NLU result has been received by the computing device from an NLU engine remotely located from the computing device, and

determining that the first ASR result and the corresponding NLU result should be added to the at least one data structure when the number of times the corresponding NLU result has been received by the computing device exceeds a threshold value.

16. The computing device of claim 14 ,

wherein the audio received by the input interface comprises a third audio portion,

wherein the ASR engine generates a third ASR result based, at least in part, on at least a portion of the third audio portion, and

wherein the at least one processor is further programmed to:

identify an ASR output included in the at least one data structure, the identified ASR output corresponding to the third ASR result, and

submit a NLU result corresponding to the identified ASR output included in the at least one data structure to the speech-enabled application to enable the speech-enabled application to perform an action based on the submitted NLU result.

17. The computing device of claim 16 , wherein the at least one processor is programmed to submit the NLU result corresponding to the identified ASR output included in the at least one data structure without sending a request for remote NLU processing of the third audio portion to an NLU engine remotely located from the computing device.

18. A method, comprising:

receiving, by an input interface of a computing device, audio comprising speech from a user of the computing device;

detecting, by an automatic speech recognition (ASR) engine of the computing device, an end of speech in a first audio portion of the received audio;

generating, by the ASR engine, an ASR result based, at least in part, on a portion of the first audio portion prior to the detected end of speech;

determining, by at least one programmed processor of the computing device, whether a valid action can be performed by a speech-enabled application installed on the computing device using the ASR result;

instructing, by the at least one programmed processor, the ASR engine to process a second audio portion of the received audio, recorded after the detected end of speech of the first audio portion, when it is determined that a valid action cannot be performed by the speech-enabled application using the ASR result; and

processing, by the ASR engine, the second audio portion in addition to the first audio portion when instructed by the at least one programmed processor.

19. A non-transitory computer-readable storage medium encoded with a plurality of instructions that, when executed by a computing device, performs a method, the method comprising:

receiving audio comprising speech from a user of the computing device;

detecting an end of speech in a first audio portion of the received audio;

generating an ASR result based, at least in part, on a portion of the first audio portion prior to the detected end of speech;

determining, using the ASR result, whether a valid action can be performed by a speech-enabled application installed on the computing device; and

processing the first audio portion and a second audio portion of the received audio, recorded after the detected end of speech of the first audio portion, when it is determined that a valid action cannot be performed by the speech-enabled application using the ASR result.