Weighted finite state transducer frameworks for conversational AI systems and applications
Systems and methods provide for a machine learning system to train a machine learning model to output a penalty-free emission when processing an auditory input. For example, as the system generates paths through a probability lattice, one or more paths may include a penalty-free emission that skips at least one frame associated with the probability lattice, but that does not add a cost to a final path cost. The use of the penalty-free emissions may be represented through one or more graphical representations used for training in order to develop loss functions for models. One or more of these frameworks may be incorporated into automatic speech recognition pipelines to improve training while also reducing coding requirements to simplify debugging operations.
1 . A processor comprising:
one or more circuits configured to perform, using a neural network (NN), one or more automatic speech recognition (ASR) operations with respect to a sequence of audio frames, the operations using at least a three-dimensional (3D) tensor converted from a four-dimensional (4D) tensor by combination of an audio dimension and a text dimension, the one or more circuits configured to, responsive to an output of the NN indicating an emulated blank for at least one audio frame of the sequence of audio frames, omit at least one audio frame from processing without affecting a loss score.
2 . The processor of claim 1 , wherein the processor is comprised at least one of:
a system for performing simulation operations;
a system for performing simulation operations to test or validate autonomous machine applications;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for rendering graphical output;
a system for performing deep learning operations;
a system implemented using an edge device;
a system for generating or presenting virtual reality (VR) content;
a system for generating or presenting augmented reality (AR) content;
a system for generating or presenting mixed reality (MR) content;
a system incorporating one or more Virtual Machines (VMs);
a system for performing operations for a conversational AI application;
a system for performing operations for a generative AI application;
a system for performing operations using a language model;
a system implemented at least partially in a data center;
a system for performing hardware testing using simulation;
a system for synthetic data generation;
a collaborative content creation platform for 3D assets; or
a system implemented at least partially using cloud computing resources.
3 . The processor of claim 1 , wherein the NN includes a recurrent neural network transducer (RNN-T).
4 . The processor of claim 1 , wherein the NN is trained using at least one path through a probability lattice.
5 . The processor of claim 4 , wherein the probability lattice is formed based, at least, on the 3D tensor.
6 . The processor of claim 5 , wherein the one or more circuits are further to reshape the 4D tensor into the 3D tensor so that the combination of the audio dimension and text dimension are represented by a single value.
7 . The processor of claim 1 , wherein the one or more circuits are further to generate a combined lattice representation, for an utterance associated with the sequence of audio frames, including a unit schema and a time schema.
8 . The processor of claim 7 , wherein one or more arcs of the unit schema correspond to an epsilon emission with a weight of zero.
9 . A method, comprising:
receiving a sequence of audio frames;
converting a four-dimensional (4D) tensor into a three-dimensional (3D) tensor at least by combining an audio dimension and a text dimension;
performing, using a neural network (NN), one or more automatic speech recognition (ASR) operations using the sequence of audio frames and the 3D tensor; and
generating, using the NN, an output including at least one audio frame of the sequence of audio frames corresponding to an emulated blank, wherein the emulated blank causes an audio frame in the sequence of audio frames to be omitted from processing, by the NN, without affecting a loss score.
10 . The method of claim 9 , wherein the NN includes a recurrent neural network transducer (RNN-T).
11 . The method of claim 9 , wherein the NN is trained using at least one path through a probability lattice.
12 . The method of claim 11 , wherein the probability lattice is formed based, at least, on the 3D tensor.
13 . The method of claim 12 , further comprising:
reshaping the 4D tensor into the 3D tensor so that the combined audio dimension and text dimension are represented by a single value.
14 . The method of claim 9 , further comprising:
generating a combined lattice representation, for an utterance associated with the sequence of audio frames, including a unit schema and a time schema.
15 . The method of claim 14 , wherein one or more arcs of the unit schema correspond to an epsilon emission with a weight of zero.
16 . A system comprising:
at least one processor to:
receive a sequence of audio frames;
convert a four-dimensional (4D) tensor into a three-dimensional (3D) tensor by combining an audio dimension and a text dimension;
perform, using a neural network (NN), one or more automatic speech recognition (ASR) operations using the sequence of audio frames and the 3D tensor; and
generate, using the NN, an output including at least one audio frame of the sequence of audio frames corresponding to an emulated blank, wherein the emulated blank causes an audio frame in the sequence of audio frames to be omitted from processing, by the NN, without affecting a loss score.
17 . The system of claim 16 , wherein the system comprises at least one of:
a system for performing simulation operations;
a system for performing simulation operations to test or validate autonomous machine applications;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for rendering graphical output;
a system for performing deep learning operations;
a system implemented using an edge device;
a system for generating or presenting virtual reality (VR) content;
a system for generating or presenting augmented reality (AR) content;
a system for generating or presenting mixed reality (MR) content;
a system incorporating one or more Virtual Machines (VMs);
a system for performing operations for a conversational AI application;
a system for performing operations for a generative AI application;
a system for performing operations using a language model;
a system implemented at least partially in a data center;
a system for performing hardware testing using simulation;
a system for synthetic data generation;
a collaborative content creation platform for 3D assets; or
a system implemented at least partially using cloud computing resources.
18 . The system of claim 16 , wherein the NN includes a recurrent neural network transducer (RNN-T).
19 . The system of claim 16 , wherein the NN is trained using at least one path through a probability lattice.
20 . The system of claim 19 , wherein the probability lattice is formed based, at least, on the 3D tensor.