Textual knowledge transfer for improved speech recognition and understanding
Systems, computer-implemented methods, and computer program products to facilitate fine-grained textual knowledge transfer to improve speech recognition and understanding are provided. According to an embodiment, a system can comprise a processor that executes components stored in memory. The computer executable components comprise deriving component that can derive one or more speech-based embeddings from an utterance via a speech encoder. The computer executable components can comprise a cross-attention component that can align, at a token level, one or more large language model (LLM) based sentence embeddings with the one or more speech-based embeddings. The computer executable components can comprise a loss component that can combine an alignment loss and an automatic speech recognition (ASR) loss.
1. A system comprising:
a processor that executes computer-executable components stored in a non-transitory computer-readable memory, wherein the computer-executable components comprise:
a deriving component that derives one or more speech-based embeddings from an utterance via a speech encoder;
a cross-attention component that aligns, at a token level, one or more Large Language Model (LLM) based sentence embeddings with the one or more speech-based embeddings;
a loss component that combines an alignment loss and an Automatic Speech Recognition (ASR) loss; and
a training component that trains an ASR system with an end-to-end framework using the loss component and the cross-attention component to produce one or more enriched embeddings.
2. The system of claim 1 , wherein the cross-attention component determines a contrastive loss between the one or more LLM based sentence embeddings and the one or more speech-based embeddings.
3. The system of claim 1 , wherein the cross-attention component uses non-contextual (NC) embeddings as queries to align the one or more LLM based sentence embeddings with the one or more speech-based embeddings.
4. The system of claim 1 , wherein the ASR system is adapted to perform a Spoken Language Understanding (SLU) task.
5. The system of claim 4 , wherein the cross-attention component creates speech-based embeddings that are aligned with the one or more LLM based sentence embeddings, and the speech-based embeddings are fused with one or more ASR based embeddings.
6. The system of claim 5 , wherein the cross-attention component creates a speech-based summary token to approximate the one or more LLM based sentence embeddings to determine an SLU loss.
7. The system of claim 6 , wherein a gating mechanism integrates the speech-based summary token along with other embeddings of an end-to-end ASR model to improve a final training loss.
8. A computer-implemented method comprising:
deriving, by a system operatively coupled to a processor, one or more speech-based embeddings from an utterance via a speech encoder;
aligning at a token level, by the system, one or more Large Language Model (LLM) based sentence embeddings with the one or more speech-based embeddings;
combining, by the system, an alignment loss and an Automatic Speech Recognition (ASR) loss; and
training, by the system, an ASR system with an end-to-end framework using alignment loss, the ASR loss, and the one or more LLM based sentence embeddings aligned with the one or more speech-based embeddings to produce one or more enriched embeddings.
9. The computer-implemented method of claim 8 , further comprising:
determining, by the system, a contrastive loss between the one or more LLM based sentence embeddings and the one or more speech-based embeddings.
10. The computer-implemented method of claim 8 , further comprising:
using, by the system, non-contextual embeddings as queries to align the one or more LLM based sentence embeddings with the one or more speech-based embeddings.
11. The computer-implemented method of claim 8 , further comprising:
adapting, by the system, the ASR system to perform a Spoken Language Understanding (SLU) task.
12. The computer-implemented method of claim 11 , further comprising:
creating, by the system, speech-based embeddings aligned with the one or more LLM based sentence embeddings, and
fusing, by the system, the speech-based embeddings with one or more ASR based embeddings.
13. The computer-implemented method of claim 12 , further comprising:
creating, by the system, a speech-based summary token to approximate the one or more LLM based sentence embeddings to determine an SLU loss.
14. The computer-implemented method of claim 13 , further comprising:
integrating, by the system, the speech-based summary token along with other embeddings of an end-to-end ASR model to improve a final training loss.
15. A computer program product comprising a non-transitory computer readable memory having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:
derive, by the processor, one or more speech-based embeddings from an utterance via a speech encoder;
align at a token level, by the processor, one or more Large Language Model (LLM) based sentence embeddings with the one or more speech-based embeddings;
combine, by the processor, an alignment loss and an Automatic Speech Recognition (ASR) loss; and
train, by the processor, an Automatic Speech Recognition (ASR) system, with an end-to-end framework using the alignment loss, the ASR loss, and the one or more LLM based sentence embeddings aligned with the one or more speech-based embeddings to produce one or more enriched embeddings.
16. The computer program product of claim 15 , wherein the program instructions are further executable to cause the processor to:
determine, by the processor, a contrastive loss between the one or more LLM based sentence embeddings and the one or more speech-based embeddings.
17. The computer program product of claim 15 , wherein the program instructions are further executable to cause the processor to:
use, by the processor, non-contextual embeddings as queries to align the one or more LLM based sentence embeddings with the one or more speech-based embeddings.
18. The computer program product of claim 15 , wherein the program instructions are further executable to cause the processor to:
adapt, by the processor, the ASR system to perform a Spoken Language Understanding (SLU) task; and
create, by the processor, speech-based embeddings aligned with the one or more LLM based sentence embeddings, and
fusing, by the system, the speech-based embeddings with one or more ASR based embeddings.
19. The computer program product of claim 18 , wherein the program instructions are further executable to cause the processor to:
create, by the processor, a speech-based summary token to approximate the one or more LLM based sentence embeddings to determine an SLU loss.
20. The computer program product of claim 19 , wherein the program instructions are further executable to cause the processor to:
integrate, by the processor, the speech-based summary token along with other embeddings of an end-to-end ASR model to improve a final training loss.