Granted Patent B2

US 12,243,513 · App. 17/323,847 · Granted Mar 4, 2025

Inventors: Chenguang Zhu (Sammamish, WA); Nanshan Zeng (Bellevue, WA)

Assignee: Microsoft Technology Licensing, LLC

G10L15/063G06F40/279G10L13/08G10L15/02G10L15/1815G10L15/22

Abstract

Claims (63)

1. A computer implemented method for joint training a speech module with a knowledge module for natural language understanding, the method comprising:

obtaining a first knowledge graph comprising a set of entities and a set of relations between two or more entities included in the set of entities;

transforming the first knowledge graph into an acoustic knowledge graph, the acoustic knowledge graph comprising a set of acoustic data corresponding to each entity of the set of entities and each relation of the set of relations between entities;

extracting a first set of acoustic features from the acoustic knowledge graph;

training a knowledge module on the set of acoustic features to generate knowledge-based entity representations for the acoustic knowledge graph;

pre-training a speech module on a training data set comprising unlabeled speech data to understand acoustic information from speech transcriptions;

pre-training a language module on a second training data set comprising unlabeled text-based data, the language module being configured to understand semantic information from speech transcriptions;

aligning the speech module and the language module, the speech module being configured to leverage acoustic information and language information in natural language processing tasks;

obtaining a third training data set comprising paired acoustic data and transcript data;

applying the third training data set to the speech module and language module;

obtaining acoustic output embeddings from the speech module;

obtaining language output embeddings from the language module;

after pre-training the language module, extracting a set of textual features from the first knowledge graph;

training a second knowledge module on the set of textual features; and

integrating the second knowledge module with the language module before the language module and speech module are aligned;

aligning the acoustic output embeddings and the language output embeddings to a shared semantic space; and

generating an integrated knowledge-speech module to perform semantic analysis on audio data by integrating the knowledge module with the speech module, the integrated knowledge-speech module being generated by providing context information from the speech module to the knowledge module and knowledge information from the knowledge module to the speech module.

2. The method of claim 1 , wherein transforming the first knowledge graph into an acoustic knowledge graph comprises:

obtaining electronic content that describes the set of entities and the set of relations between two or more entities included in the set of entities;

extracting a second set of acoustic features from the electronic content;

transcribing the second set of acoustic features from the electronic content into text, the text representing the set of entities and the set of relations included in the first knowledge graph; and

training the knowledge module on the text and the second set of acoustic features, the knowledge module being configured to generate entity representations.

3. The method of claim 1 , wherein transforming the first knowledge graph into an acoustic knowledge graph comprises:

extracting a set of textual representations for each entity of the set of entities and each relation of the set of relations;

accessing a text-to-speech module;

applying the set of textual representations as input to the text-to-speech module; and

obtaining a set of acoustic signals corresponding to the set of textual representations as output from the text-to-speech module, each acoustic signal of the set of acoustic signals representing an entity of the set of entities included in the first knowledge graph or a relation of the set of relations included in the first knowledge graph.

4. The method of claim 1 , wherein the knowledge module of the integrated knowledge-speech module comprises a graph attention network and is configured to provide structure-aware entity embeddings for speech modeling.

5. The method of claim 1 , wherein the speech module of the integrated knowledge-speech module is further configured to produce acoustic language representations as initial embeddings for acoustic knowledge graph entities and relations.

6. The method of claim 1 , wherein integrating the speech module and knowledge module comprises projecting entity and relations output embeddings and language acoustic embeddings into a shared semantic space.

7. The method of claim 1 , wherein the speech module of the integrated knowledge-speech module comprises a first speech module and a second speech module.

8. The method of claim 7 , wherein the first speech module comprise a first set of transformer layers and the second speech module comprises a second set of transformer layers.

9. The method of claim 7 , wherein integrating the speech module and knowledge module further comprises:

obtaining a first set of acoustic language embeddings from the first speech module; and

applying the first set of acoustic language embeddings as input to the second speech module and the knowledge module.

10. The method of claim 9 , further comprising:

obtaining a first set of acoustic entity embeddings from the knowledge module;

applying the first set of acoustic entity embeddings as input to the second speech module; and

obtaining a final representation output embedding from the second speech module based on the first set of acoustic entity embeddings and the first set of acoustic language embeddings, the final representation output embedding including acoustic and knowledge information.

11. The method of claim 1 , wherein each entity of the set of entities is represented by entity description audio data to describe a concept and meaning of the entities, and each relation of the set of relations being represented by relation description audio data to describe each relation between two or more entities.

12. The method of claim 1 , further comprising:

optimizing the integrated knowledge-speech module by performing one or more following training tasks: entity category prediction, relation type prediction, masked token prediction, or masked entity prediction.

13. A computing system configured for joint training a speech module with a knowledge module for natural language understanding, the computing system comprising:

one or more processors; and

one or more hardware storage devices that store computer-executable instructions that are structured to be executed by the one or more processors to cause the computing system to at least:

obtain a first knowledge graph comprising a set of entities and a set of relations between two or more entities included in the set of entities;

transform the first knowledge graph into an acoustic knowledge graph, the acoustic knowledge graph comprising acoustic data corresponding to each entity of the set of entities and each relation of the set of relations between entities;

extract a set of acoustic features from the acoustic knowledge graph;

train a knowledge module on the set of acoustic features to generate knowledge-based entity representations for the acoustic knowledge graph;

pre-train a speech module on a training data set comprising unlabeled speech data to understand acoustic information from speech transcriptions;

pre-train a language module on a second training data set comprising unlabeled text-based data, the language module being configured to understand semantic information from speech transcriptions; and

after pre-training the language module, extracting a set of textual features from the first knowledge graph;

training a second knowledge module on the set of textual features;

integrating the second knowledge module with the language module before the language module and speech module are aligned;

align the speech module and the language module, the speech module being configured to leverage acoustic information and language information in natural language processing tasks; and

generate an integrated knowledge-speech module to perform semantic analysis on audio data by integrating the knowledge module with the speech module, the integrated knowledge-speech module being trained by providing context information from the speech module to the knowledge module and knowledge information from the knowledge module to the speech module.

14. A computer implemented method for joint training a speech module with a knowledge module for natural language understanding, the method comprising:

obtaining a first knowledge graph comprising a set of entities and a set of relations between two or more entities included in the set of entities;

transforming the first knowledge graph into an acoustic knowledge graph, the acoustic knowledge graph comprising a set of acoustic data corresponding to each entity of the set of entities and each relation of the set of relations between entities;

extracting a first set of acoustic features from the acoustic knowledge graph;

training a knowledge module on the set of acoustic features to generate knowledge-based entity representations for the acoustic knowledge graph;

pre-training a speech module on a training data set comprising unlabeled speech data to understand acoustic information from speech transcriptions, the speech module comprising a first set of transformer layers in a first speech module and a second set of transformer layers in a second speech module; and

generating an integrated knowledge-speech module to perform semantic analysis on audio data by integrating the knowledge module with the speech module, the integrated knowledge-speech module being generated by at least obtaining a first set of acoustic language embeddings from the first speech module and applying the first set of acoustic language embeddings as input to the second speech module and the knowledge module, providing context information from the speech module to the knowledge module and knowledge information from the knowledge module to the speech module, obtaining a first set of acoustic entity embeddings from the knowledge module, applying the first set of acoustic entity embeddings as input to the second speech module, and obtaining a final representation output embedding from the second speech module based on the first set of acoustic entity embeddings and the first set of acoustic language embeddings, the final representation output embedding including acoustic and knowledge information.

Assignments (1)

Continuity (2)

References Cited (129)

Cited By (2)

US 12,518,741 US 12,711,308