Speech-to-text processing assisted with language models for conversational AI systems and applications
Disclosed are apparatuses, systems, and techniques that implement training and deployment of speech-augmented language models for efficient capturing and processing of speech inputs. The techniques include processing, using a speech model, an audio input in a first language to generate a first portion of an input into a language model (LM). A second portion of the input into the LM includes represents a text context associated with the audio input. The techniques further include receiving, from the LM, an output that includes a speech-to-text conversion of the audio input.
1 . A method comprising:
processing, using a speech model that comprises a neural network (NN) of artificial neurons, an audio input in a first language to generate one or more speech embeddings;
processing, using a speech adapter NN, the one or more speech embeddings to generate a first set of one or more tokens in a format associated with a language model (LM);
obtaining
a second set of one or more tokens representing a text context associated with the audio input;
obtaining a third set of one or more tokens identifying a type of a speech-to-text conversion to be performed using the LM;
aggregating the first set of one or more tokens, the second set of one or more tokens, and the third set of one or more tokens to obtain an aggregated set of tokens; and
processing, using the LM, the aggregated set of tokens to obtain an output including a speech-to-text conversion of the audio input.
2 . The method of claim 1 , wherein the text context comprises:
one or more keywords associated with the audio input.
3 . The method of claim 1 , wherein the text context is received from a speaker that generated the audio input.
4 . The method of claim 1 , further comprising:
identifying a subject area associated with the audio input; and
assembling the text context from one or more entries that are stored in association with the identified subject area.
5 . The method of claim 4 , wherein the identifying the subject area associated with the audio input comprises:
receiving a preliminary output of the LM, the preliminary output generated in response to a preliminary input generated using the audio input.
6 . The method of claim 1 , wherein the speech-to-text conversion comprises at least one of:
a transcription of the audio input in the first language, or
a translation of the audio input into a second language.
7 . The method of claim 1 , wherein processing the aggregated set of tokens comprises:
processing, using an LM adapter NN, a copy of the aggregated set of tokens to generate an adapter output; and
combining the adapter output with an intermediate output of the LM.
8 . The method of claim 1 , wherein the NN of artificial neurons of the speech model comprises one or more conformer blocks.
9 . The method of claim 1 , wherein the adapter NN is trained after training of the LM.
10 . A system comprising:
one or more processing units to generate, using a language model (LM) comprising an artificial neural network (ANN), a speech-to-text conversion of an audio input, the speech-to-text conversion generated and based at least on (i) a first processing, by the LM, of an aggregated input comprising a first portion generated based at least on a speech model processing the audio input and a second portion of the input representing contextual information associated with the audio input, (ii) a second processing, by an LM adapter ANN, of the aggregated input, and (iii) aggregation of outputs of the first processing and the second processing.
11 . The system of claim 10 , wherein the contextual information comprises:
one or more keywords associated with the audio input, and wherein the aggregated input further comprises:
a third portion identifying a type of the speech-to-text conversion to be performed using the LM.
12 . The system of claim 10 , wherein to process the audio input to generate the first portion of the input into the LM, the one or more processing units are to:
convert, using a speech adapter ANN, an output of the speech model into one or more tokens of a format associated with the LM.
13 . The system of claim 10 , wherein the system is comprised in at least one of:
an in-vehicle infotainment system for an autonomous or semi-autonomous machine;
a system for performing one or more simulation operations;
a system for performing one or more digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing one or more deep learning operations;
a system implemented using an edge device;
a system for generating or presenting at least one of virtual reality content, mixed reality content, or augmented reality content;
a system implemented using a robot;
a system for performing one or more conversational AI operations;
a system implementing one or more large language models (LLMs);
a system implementing one or more language models;
a system for performing one or more generative AI operations;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
14 . One or more processors to:
process, using a speech model that comprises a neural network (NN) of artificial neurons, an audio input in a first language to generate one or more speech embeddings;
process, using a speech adapter NN, the one or more speech embeddings to generate a first set of one or more tokens in a format associated with a language model (LM);
obtain a second set of one or more tokens representing a text context associated with the audio input;
obtain a third set of one or more tokens identifying a type of a speech-to-text conversion to be performed using the LM;
aggregate the first set of one or more tokens, the second set of one or more tokens, and the third set of one or more tokens to obtain an aggregated set of tokens; and
process, using the LM, the aggregated set of tokens to obtain an output including a speech-to-text conversion of the audio input.
15 . The one or more processors of claim 14 , wherein the text context comprises:
one or more keywords associated with the audio input.
16 . The one or more processors of claim 14 , wherein the one or more processors are further to:
identify a subject area associated with the audio input; and
assemble the text context from one or more entries that are stored in association with the identified subject area.
17 . The one or more processors of claim 14 , wherein the speech-to-text conversion comprises at least one of:
a transcription of the audio input in the first language, or
a translation of the audio input into a second language.
18 . The one or more processors of claim 14 , wherein to process the aggregated set of tokens, the one or more processors are further to:
process, using an LM adapter NN, a copy of the aggregated set of tokens to generate an adapter output; and
combine the adapter output with an intermediate output of the LM.
19 . The one or more processors of claim 14 , wherein the adapter NN is trained after training of the LM.