System and method for command fulfillment without wake word

1 . A method comprising:

obtaining, using at least one processor of an electronic device, an audio input; providing, using the at least one processor, at least a portion of the audio input to a frame-level detector model;

obtaining, using the at least one processor, a first output of the frame-level detector model including frame-level predictions associated with at least the portion of the audio input;

determining, based on the frame-level predictions associated with the at least the portion of the audio input, at least one chunked audio frame, corresponding to one or more words predicted to be one or more target domain-specific words by the frame-level detector model;

providing, using the at least one processor, the at least one chunked audio frame to a word-level verifier model, wherein the at least one chunked audio frame corresponding to one or more words predicted to be the one or more target domain-specific words is separated from other frames of the audio input that include audio data with pre-speech or post-speech portions, such that only the at least one chunked audio frame is provided to the word-level verifier model;

obtaining, using the at least one processor, a second output of the word-level verifier model including word-level probabilities associated with the at least one chunked audio frame independently from the frame-level predictions, wherein the word-level verifier model receives and processes only the at least one chunked audio frame based on the separation of the at least one chunked audio frame from the other frames of the audio input that include the audio data with pre-speech or post-speech portions; and

instructing, using the at least one processor, performance of automatic speech recognition on the audio input based on the word-level probabilities associated with the at least one chunked audio frame.

2 . The method of claim 1 , wherein each of the word-level probabilities received from the word-level verifier model indicates a likelihood that the one or more words predicted to be the one or more target domain-specific words by the frame-level detector model is in one of a plurality of classes.

3 . The method of claim 1 , further comprising:

providing, using the at least one processor, labeled audio frames obtained from the frame-level detector model to a false-trigger mitigation model;

providing, using the at least one processor, an additional set of audio frames from the audio input, the additional set of audio frames including audio before or after the labeled audio frames in the audio input; and

obtaining, using the at least one processor, a third output of the false-trigger mitigation model including a prediction of whether the audio input is related to a target domain.

4 . The method of claim 3 , further comprising:

determining, using the at least one processor, a score based on a combination of the second output and the third output; and

instructing, using the at least one processor, the performance of the automatic speech recognition based on a determination that the score is greater than a threshold.

5 . The method of claim 3 , further comprising:

providing, using the at least one processor, the at least one chunked audio frame to a speaker verification model;

obtaining, using the at least one processor, a fourth output of the speaker verification model including a prediction of whether the audio input was provided by an enrolled speaker;

determining, using the at least one processor, a score based on a combination of the second output, the third output, and the fourth output; and

instructing, using the at least one processor, the performance of the automatic speech recognition based on a determination that the score is greater than a threshold.

6 . The method of claim 1 , further comprising:

training, using the at least one processor, the frame-level detector model and the word-level verifier model;

wherein the training comprises:

providing, using the at least one processor, pairs of audio inputs to an auxiliary task model, each of the pairs of audio inputs comprising (i) a first audio input including a chunked audio sample and (ii) a second audio input including either a positive audio sample or a negative audio sample;

determining, for each of the pairs of audio inputs, using the at least one processor, a contrastive loss between a first output of the auxiliary task model associated with the first audio input and a second output of the auxiliary task model associated with the second audio input;

generating, using the at least one processor, the frame-level detector model by modifying one or more layers of the auxiliary task model to perform a first primary task; and

generating, using the at least one processor, the word-level verifier model by modifying the one or more layers of the auxiliary task model to perform a second primary task.

7 . The method of claim 6 , wherein:

each positive audio sample includes one of: (i) a same chunked audio sample as the chunked audio sample of the corresponding first audio input or (ii) the same chunked audio sample as the chunked audio sample of the corresponding first audio input augmented with at least one of noise and reverberations; and

each negative audio sample includes one of: (i) the noise used for augmentation, (ii) a different chunked audio sample than the chunked audio sample of the corresponding first audio input, or (iii) the different chunked audio sample than the chunked audio sample of the corresponding first audio input augmented with at least one of noise and reverberations.

8 . The method of claim 6 , further comprising:

further training, using the at least one processor, each of the frame-level detector model and the word-level verification model to perform the first primary task and the second primary task, respectively, wherein each of the frame-level detector model and the word-level verifier model includes one or more initial layers from the auxiliary task model,

wherein further training each of the frame-level detector model and the word-level verification model comprises one of:

fine-tuning all layers of the frame-level detector model and all layers of the word-level verification model; or

training the one or more layers of the frame-level detector model modified to perform the first primary task and training the one or more layers of the word-level verifier model modified to perform the second primary task, wherein the one or more initial layers of the auxiliary task model included in each of the frame-level detector model and the word-level verification model remain fixed.

9 . The method of claim 6 , wherein:

the first primary task includes outputting labels associated with sets of audio frames for an utterance, each of the labels indicating whether one of the sets of audio frames includes the one or more target domain-specific words; and

the second primary task includes receiving chunked audio frames determined by the frame-level detector model to include the target domain-specific word and outputting word-level probabilities associated with the chunked audio frames, each of the word-level probabilities indicating a likelihood that the target domain-specific word in the chunked audio frames is in one of a plurality of classes.

10 . An apparatus comprising:

at least one processing device configured to:

obtain an audio input;

provide at least a portion of the audio input to a frame-level detector model;

obtain a first output of the frame-level detector model including frame-level predictions associated with at least the portion of the audio input;

determine, based on the frame-level predictions associated with the at least the portion of the audio input, at least one chunked audio frame, corresponding to one or more words predicted to be one or more target domain-specific words by the frame-level detector model;

provide the at least one chunked audio frame to a word-level verifier model, wherein the at least one chunked audio frame corresponding to one or more words predicted to be the one or more target domain-specific words is separated from other frames of the audio input that include audio data with pre-speech or post-speech portions, such that only the at least one chunked audio frame is provided to the word-level verifier model;

obtain a second output of the word-level verifier model including word-level probabilities associated with the at least one chunked audio frame independently from the frame-level predictions, wherein the word-level verifier model receives and processes only the at least one chunked audio frame based on the separation of the at least one chunked audio frame from the other frames of the audio input that include the audio data with pre-speech or post-speech portions; and

instruct performance of automatic speech recognition on the audio input based on the word-level probabilities associated with the at least one chunked audio frame.

11 . The apparatus of claim 10 , wherein each of the word-level probabilities received from the word-level verifier model indicates a likelihood that the one or more words predicted to be the one or more target domain-specific words by the frame-level detector model is in one of a plurality of classes.

12 . The apparatus of claim 10 , wherein the at least one processing device is further configured to:

provide labeled audio frames obtained from the frame-level detector model to a false-trigger mitigation model;

provide an additional set of audio frames from the audio input, the additional set of audio frames including audio before or after the labeled audio frames in the audio input; and

obtain a third output of the false-trigger mitigation model including a prediction of whether the audio input is related to a target domain.

13 . The apparatus of claim 12 , wherein the at least one processing device is further configured to:

determine a score based on a combination of the second output and the third output; and

instruct the performance of the automatic speech recognition based on a determination that the score is greater than a threshold.

14 . The apparatus of claim 12 , wherein the at least one processing device is further configured to:

provide the at least one chunked audio frame to a speaker verification model;

obtain a fourth output of the speaker verification model including a prediction of whether the audio input was provided by an enrolled speaker;

determine a score based on a combination of the second output, the third output, and the fourth output; and

instruct the performance of the automatic speech recognition based on a determination that the score is greater than a threshold.

15 . The apparatus of claim 10 , wherein the at least one processing device is further configured to:

train the frame-level detector model and the word-level verifier model, wherein the training comprises:

provide pairs of audio inputs to an auxiliary task model, each of the pairs of audio inputs comprising (i) a first audio input including a chunked audio sample and (ii) a second audio input including either a positive audio sample or a negative audio sample;

determine, for each of the pairs of audio inputs, a contrastive loss between a first output of the auxiliary task model associated with the first audio input and a second output of the auxiliary task model associated with the second audio input;

generate the frame-level detector model by modifying one or more layers of the auxiliary task model to perform a first primary task; and

generate the word-level verifier model by modifying the one or more layers of the auxiliary task model to perform a second primary task.

16 . The apparatus of claim 15 , wherein:

each positive audio sample includes one of: (i) a same chunked audio sample as the chunked audio sample of the corresponding first audio input or (ii) the same chunked audio sample as the chunked audio sample of the corresponding first audio input augmented with at least one of noise and reverberations; and

each negative audio sample includes one of: (i) the noise used for augmentation, (ii) a different chunked audio sample than the chunked audio sample of the corresponding first audio input, or (iii) the different chunked audio sample than the chunked audio sample of the corresponding first audio input augmented with at least one of noise and reverberations.

17 . The apparatus of claim 15 , wherein the at least one processing device is further configured to:

further train each of the frame-level detector model and the word-level verification model to perform the first primary task and the second primary task, respectively, wherein each of the frame-level detector model and the word-level verifier model includes one or more initial layers from the auxiliary task model,

wherein, to further train each of the frame-level detector model and the word-level verification model, the at least one processing device is configured to:

fine-tune all layers of the frame-level detector model and all layers of the word-level verification model; or

train the one or more layers of the frame-level detector model modified to perform the first primary task and training the one or more layers of the word-level verifier model modified to perform the second primary task, wherein the one or more initial layers of the auxiliary task model included in each of the frame-level detector model and the word-level verification model remain fixed.

18 . A non-transitory computer readable medium containing instructions that when executed cause at least one processor to:

obtain an audio input;

provide at least a portion of the audio input to a frame-level detector model;

obtain a first output of the frame-level detector model including frame-level predictions associated with at least the portion of the audio input;

determine, based on the frame-level predictions associated with the at least the portion of the audio input, at least one chunked audio frame corresponding to one or more words predicted to be one or more target domain-specific words by the frame-level detector model;

provide the at least one chunked audio frame to a word-level verifier model, wherein the at least one chunked audio frame corresponding to one or more words predicted to be the one or more target domain-specific words is separated from other frames of the audio input that include audio data with pre-speech or post-speech portions, such that only the at least one chunked audio frame is provided to the word-level verifier model;

obtain a second output of the word-level verifier model including word-level probabilities associated with the at least one chunked audio frame independently from the frame-level predictions, wherein the word-level verifier model receives and processes only the at least one chunked audio frame based on the separation of the at least one chunked audio frame from the other frames of the audio input that include the audio data with pre-speech or post-speech portions; and

instruct performance of automatic speech recognition on the audio input based on the word-level probabilities associated with the at least one chunked audio frame.