Processing optimization using machine learning

1. A computer-implemented method, comprising:

receiving first audio data representing a user request by a speech processing-enabled device;

generating, by an automatic speech recognition (ASR) component, first text data representing the user request;

determining a first natural language understanding (NLU) process to process the first text data to determine a first semantic interpretation of the first text data using a first set of NLU processing actions associated with the first NLU process;

determining a second NLU process for processing the first text data to determine a second semantic interpretation of the first text data using a second set of NLU processing actions associated with the second NLU process;

executing the first NLU process, comprising:

generating first intermediate NLU result data; and

generating first feature data from the first intermediate NLU result data;

executing the second NLU process in parallel with executing the first NLU process, comprising:

generating second intermediate NLU result data; and

generating second feature data from the second intermediate NLU result data;

inputting the first feature data and the second feature data into a reinforcement machine learning model;

determining, by the reinforcement machine learning model using the first feature data and the second feature data, that the first NLU process is preferred with respect to the second NLU process based on at least one of first accuracy characteristics or first latency characteristics of the first NLU process and based on at least one of second accuracy characteristics or second latency characteristics of the second NLU process;

determining at least one updated parameter of the reinforcement machine learning model using a reward function;

determining a first final result from the first NLU process; and

terminating the second NLU process.

2. The computer-implemented method of claim 1 , further comprising:

determining by a natural language understanding (NLU) arbitration component a threshold confidence score, wherein the threshold confidence score controls whether at least one of the processing of the first NLU process or the second NLU process is interrupted in response to the determining that the first NLU process is preferred with respect to the second NLU process by the reinforcement machine learning model; and

comparing the threshold confidence score to a first confidence score associated with the determination that the first NLU process is preferred with respect to the second NLU process, wherein the terminating the second NLU process is performed in response to the comparison of the threshold confidence score to the first confidence score.

3. The computer-implemented method of claim 1 , further comprising:

determining a third NLU process for processing the first text data;

executing the third NLU process in parallel with executing the first NLU process and the second NLU process, comprising:

generating third intermediate NLU result data;

generating third feature data from the third intermediate NLU result data;

determining that the third feature data is different from feature data included in a set of features used by the reinforcement machine learning model; and

determining, by the reinforcement machine learning model using the third feature data, that the third NLU process is preferred with respect to the second NLU process based on third accuracy characteristics and third latency characteristics of the third NLU process.

4. The computer-implemented method of claim 1 , further comprising:

determining that the first NLU process will experience a timeout at a first time prior to a second time at which the first final result is determined; and

extending the timeout from the first time to a third time, wherein the second time is between the first time and the third time.

5. A method comprising:

receiving first input data by a speech processing system;

executing a first natural language understanding (NLU) process for processing the first input data, wherein the first NLU process comprises a first set of NLU processing actions;

executing a second NLU process for processing the first input data, wherein the second NLU process comprises a second set of NLU processing actions different from the first set of NLU processing actions;

determining, using a first machine learning model, a first recommended action for processing the first input data by the first NLU process, wherein the first recommend action is effective to optimize at least one of the first NLU process or the second NLU process based on at least one of latency, accuracy, or performance; and

sending the first recommended action to a natural language processing component executing at least one of the first NLU process or the second NLU process.

6. The method of claim 5 , further comprising:

determining first intermediate NLU result data from processing, by the first NLU process, the first input data;

determining second intermediate NLU result data from processing, by the second NLU process, the first input data;

generating first feature data from the first intermediate NLU result data;

generating second feature data from the second intermediate NLU result data; and

inputting the first feature data and the second feature data into the first machine learning model, wherein the first recommended action is determined based at least in part on the first feature data and the second feature data.

7. The method of claim 5 , further comprising:

determining, by the first machine learning model, first data indicating that the first NLU process generates a first appropriate output from processing the first input data;

determining, by the first machine learning model, second data indicating that the second NLU process is unlikely to generate a second appropriate output from processing the first input data; and

terminating the second NLU process prior to generation of a final output by the second NLU process.

8. The method of claim 5 , further comprising:

determining that a predetermined timeout for the first NLU process will be reached prior to generating a first final output of the first NLU process;

terminating the first NLU process prior to reaching the predetermined timeout; and

generating, by the second NLU process, a second final output of the second NLU process.

9. The method of claim 5 , further comprising:

determining a third NLU process for processing the first input data in parallel with the first NLU process and the second NLU process;

determining first intermediate NLU result data from processing, by the first NLU process, the first input data;

determining second intermediate NLU result data from processing, by the second NLU process, the first input data;

generating first feature data using the first intermediate NLU result data;

generating second feature data using the second intermediate NLU result data;

determining first feedback data related to past processing of data by the third NLU process;

determining, by the first machine learning model, first data indicating that the first NLU process generates a first appropriate output from processing the first input data based at least in part on the first feature data;

determining, by the first machine learning model, second data indicating that the second NLU process generates a second appropriate output from processing the first input data based at least in part on the second feature data; and

determining, by the first machine learning model, third data indicating that the third NLU process generates a third appropriate output from processing the first input data based at least in part on the first feedback data.

10. The method of claim 5 , further comprising:

determining first intermediate NLU result data from processing, by the first NLU process, the first input data;

determining metadata associated with the first intermediate NLU result data indicating that a subsequent operation of the first NLU process is associated with a first cost; and

terminating the first NLU process prior to the subsequent operation, based at least in part on the first cost.

11. The method of claim 5 , further comprising:

determining, by the first machine learning model, first data indicating that the first NLU process generates a first appropriate output from processing the first input data;

determining, by the first machine learning model, second data indicating that the second NLU process generates a second appropriate output from processing the first input data;

determining that the second NLU process will generate second output data prior to the first NLU process generating first output data; and

generating, by the first NLU process, the first output data.

12. The method of claim 5 , further comprising:

determining a third NLU process for processing the first input data;

determining, by the first machine learning model, first data indicating that the first NLU process generates a first appropriate output from processing the first input data;

determining, by the first machine learning model, second data indicating that the second NLU process generates a second appropriate output from processing the first input data;

determining, by the first machine learning model, third data indicating that the third NLU process generates a third appropriate output from processing the first input data;

determining, based at least in part on the first data, that first output of the first NLU process is primary output;

determining, based at least in part on the second data, that second output of the second NLU process is fallback output; and

generating, based at least in part on a third confidence score, a recommendation to terminate the third NLU process prior to generating third output.

13. A computing device, comprising:

at least one processor; and

at least one non-transitory computer-readable memory storing instructions that, when executed by the at least one processor, program the at least one processor to:

receive first input data by a speech processing system;

execute a first natural language understanding (NLU) process for processing the first input data, wherein the first NLU process comprises a first set of NLU processing actions;

execute a second NLU process for processing the first input data, wherein the second NLU process comprises a second set of NLU processing actions different from the first set of NLU processing actions;

determine, using a first machine learning model, a first recommended action for processing the first input data by the first NLU process, wherein the first recommend action is effective to optimize at least one of the first NLU process or the second NLU process based on at least one of latency, accuracy, or performance; and

send the first recommended action to a natural language processing component executing at least one of the first NLU process or the second NLU process.

14. The computing device of claim 13 , the at least one non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor program the at least one processor to:

determine first intermediate NLU result data from processing, by the first NLU process, the first input data;

determine second intermediate NLU result data from processing, by the second NLU process, the first input data;

generate first feature data from the first intermediate NLU result data;

generate second feature data from the second intermediate NLU result data; and

input the first feature data and the second feature data into the first machine learning model, wherein the first recommended action is determined based at least in part on the first feature data and the second feature data.

15. The computing device of claim 13 , the at least one non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor program the at least one processor to:

determine, by the first machine learning model, first data indicating that the first NLU process generates a first appropriate output from processing the first input data;

determine, by the first machine learning model, second data indicating that the second NLU process is unlikely to generate a second appropriate output from processing the first input data; and

terminate the second NLU process prior to generation of a final output by the second NLU process.

16. The computing device of claim 13 , the at least one non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor program the at least one processor to:

determine that a predetermined timeout for the first NLU process will be reached prior to generating a first final output of the first NLU process;

terminate the first NLU process prior to reaching the predetermined timeout; and

generate, by the second NLU process, a second final output of the second NLU process.

17. The computing device of claim 13 , the at least one non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor program the at least one processor to:

determine a third NLU process for processing the first input data in parallel with the first NLU process and the second NLU process;

determine first intermediate NLU result data from processing, by the first NLU process, the first input data;

determine second intermediate NLU result data from processing, by the second NLU process, the first input data;

generate first feature data using the first intermediate NLU result data;

generate second feature data using the second intermediate NLU result data;

determine first feedback data related to past processing of data by the third NLU process;

determine, by the first machine learning model, first data indicating that the first NLU process generates a first appropriate output from processing the first input data based at least in part on the first feature data;

determine, by the first machine learning model, second data indicating that the second NLU process generates a second appropriate output from processing the first input data based at least in part on the second feature data; and

determine, by the first machine learning model, third data indicating that the third NLU process generates a third appropriate output from processing the first input data based at least in part on the first feedback data.

18. The computing device of claim 13 , the at least one non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor program the at least one processor to:

determine first intermediate NLU result data from processing, by the first NLU process, the first input data;

determine metadata associated with the first intermediate NLU result data indicating that a subsequent operation of the first NLU process is associated with a first cost; and

terminate the first NLU process prior to the subsequent operation, based at least in part on the first cost.

19. The computing device of claim 13 , the at least one non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor program the at least one processor to:

determine, by the first machine learning model, first data indicating that the first NLU process generates a first appropriate output from processing the first input data;

determine, by the first machine learning model, second data indicating that the second NLU process generates a second appropriate output from processing the first input data;

determine that the second NLU process will generate second output data prior to the first NLU process generating first output data; and

generate, by the first NLU process, the first output data.

20. The computing device of claim 13 , the at least one non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor program the at least one processor to:

determine a third NLU process for processing the first input data;

determine, by the first machine learning model, first data indicating that the first NLU process generates a first appropriate output from processing the first input data;

determine, by the first machine learning model, second data indicating that the second NLU process generates a second appropriate output from processing the first input data;

determine, by the first machine learning model, third data indicating that the third NLU process generates a third appropriate output from processing the first input data;

determine, based at least in part on the first data, that first output of the first NLU process is primary output;

determine, based at least in part on the second data, that second output of the second NLU process is fallback output; and

generate, based at least in part on a third confidence score, a recommendation to terminate the third NLU process prior to generating third output.