System and method for knowledge-based audio-text modeling via automatic multimodal graph construction
Knowledge-based audio-text modeling via automatic multimodal graph construction is performed. An audio dataset is received, the audio dataset including clips of audio data, wherein each of the clips of the audio data is paired with corresponding metadata descriptive of the audio contents of the respective clip of the audio data. Graph nodes of interest are identified from a sematic network, the graph nodes being descriptive of semantics of the knowledge domain of the contents of the audio dataset. A large language model (LLM) is utilized for categorizing the metadata into the graph nodes and for inferring supplemental data for the graph nodes for which there is no metadata, producing an extracted knowledge graph. The extracted knowledge graph is validated utilizing the LLM to perform relation verification of edges between the graph nodes of the extracted knowledge graph, thereby mitigating hallucination effects in the categorizing and inferring of the supplemental data.
1 . A method for knowledge-based audio-text modeling via automatic multimodal graph construction, comprising:
receiving an audio dataset, the audio dataset including clips of audio data, wherein each of the clips of the audio data is paired with corresponding metadata descriptive of audio contents of the respective clip;
identifying graph nodes of interest from a semantic network, the graph nodes being descriptive of semantics of a knowledge domain of the audio dataset;
utilizing a large language model (LLM) for categorizing the metadata into the graph nodes and for inferring supplemental data for the graph nodes for which there is no metadata, producing an extracted knowledge graph;
validating the extracted knowledge graph utilizing the LLM to perform relation verification of edges between the graph nodes of the extracted knowledge graph, thereby mitigating hallucination effects in the categorizing and inferring of the supplemental data; and
utilizing the knowledge graph, as validated, for a downstream application, including providing the knowledge graph to an audio generation model as a prompt to generate audio data based on the knowledge graph.
2 . The method of claim 1 , wherein the metadata includes human annotations describing the audio contents of the respective clips of the audio data.
3 . The method of claim 1 , wherein the metadata includes machine-learned labels, attributes, and/or other forms of recognition outcomes inferred from audio and/or speech data using one or more machine learning models.
4 . The method of claim 1 , wherein the graph nodes of interest are one or more of: defined based on user knowledge of the knowledge domain, queried from the semantic network as graph nodes describing semantics of the knowledge domain; extracted from a database of domain knowledge; received from the LLM responsive to a prompt for relevant graph nodes for the knowledge domain.
5 . The method of claim 1 , wherein inferring the supplemental data includes receiving the supplemental data from the LLM responsive to a prompt for requesting the LLM to infer content for names of the graph nodes for which there is no metadata available.
6 . The method of claim 1 , wherein the downstream application includes an audio classification application using the knowledge graph for sound event detection and/or audio tagging.
7 . The method of claim 1 , wherein the downstream application includes an audio captioning application using the knowledge graph for audio retrieval.
8 . The method of claim 1 , wherein the downstream application includes representing the knowledge graph as an adjacency matrix to perform multimodal graph representation learning.
9 . The method of claim 1 , wherein the downstream application includes using the knowledge graph to define knowledge-based clusters for contrastive learning.
10 . The method of claim 1 , wherein the downstream application includes using the knowledge graph to curate controllable prompts, captions, and/or descriptive contents for building knowledge-guided generative models.
11 . The method of claim 1 , further comprising:
converting the knowledge graph into a textual representation; and
providing the textual representation as a textual prompt to a text-to-audio model to generate the audio data.
12 . The method of claim 1 , further comprising:
displaying the knowledge graph via a knowledge graph editor;
receiving user input via the knowledge graph editor to substitute at least one node of the knowledge graph with a different node; and
providing the knowledge graph, as substituted, to the audio generation model, thereby modifying a characteristic of the audio data generated by the audio generation model.
13 . A system for knowledge-based audio-text modeling via automatic multimodal graph construction, comprising:
one or more hardware computing devices configured to:
receive an audio dataset, the audio dataset including clips of audio data, wherein each of the clips of the audio data is paired with corresponding metadata descriptive of the audio contents of the respective clip;
identify graph nodes of interest from a semantic network, the graph nodes being descriptive of semantics of a knowledge domain of the audio dataset;
utilize a large language model (LLM) for categorizing the metadata into the graph nodes and for inferring supplemental data for the graph nodes for which there is no metadata, producing an extracted knowledge graph;
validate the extracted knowledge graph utilizing the LLM to perform relation verification of edges between the graph nodes of the extracted knowledge graph, thereby mitigating hallucination effects in the categorizing and inferring of the supplemental data; and
utilize the knowledge graph, as validated, for a downstream application, including providing the knowledge graph to an audio generation model as a prompt to generate audio data based on the knowledge graph.
14 . The system of claim 13 , wherein the metadata includes human annotations describing the audio contents of the respective clips of the audio data.
15 . The system of claim 13 , wherein the metadata includes machine-learned labels, attributes, and/or other forms of recognition outcomes inferred from audio and/or speech data using one or more machine learning models.
16 . The system of claim 13 , wherein the graph nodes of interest are one or more of: defined based on user knowledge of the knowledge domain, queried from the semantic network as graph nodes describing semantics of the knowledge domain; extracted from a database of domain knowledge; received from the LLM responsive to a prompt for relevant graph nodes for the knowledge domain.
17 . The system of claim 13 , wherein inferring the supplemental data includes receiving the supplemental data from the LLM responsive to a prompt for requesting the LLM to infer content for names of the graph nodes for which there is no metadata available.
18 . The system of claim 13 , wherein the downstream application includes an audio classification application using the knowledge graph for sound event detection and/or audio tagging.
19 . The system of claim 13 , wherein the downstream application includes an audio captioning application using the knowledge graph for audio retrieval.
20 . The system of claim 13 , wherein the downstream application includes representing the knowledge graph as an adjacency matrix to perform multimodal graph representation learning.
21 . The system of claim 13 , wherein the downstream application includes using the knowledge graph to define knowledge-based clusters for contrastive learning.
22 . The system of claim 13 , wherein the downstream application includes using the knowledge graph to curate controllable prompts/captions/descriptive contents for building knowledge-guided generative models.
23 . A non-transitory computer-readable medium comprising instructions for a knowledge-based audio-text modeling via automatic multimodal graph construction that, when executed by one or more hardware computing devices cause the one or more hardware computing devices to perform operations including to:
receive an audio dataset, the audio dataset including clips of audio data, wherein each of the clips of the audio data is paired with corresponding metadata descriptive of the audio contents of the respective clip;
identify graph nodes of interest from a semantic network, the graph nodes being descriptive of semantics of a knowledge domain of the audio dataset;
utilize a large language model (LLM) for categorizing the metadata into the graph nodes and for inferring supplemental data for the graph nodes for which there is no metadata, producing an extracted knowledge graph;
validate the extracted knowledge graph utilizing the LLM to perform relation verification of edges between the graph nodes of the extracted knowledge graph, thereby mitigating hallucination effects in the categorizing and inferring of the supplemental data; and
utilize the knowledge graph, as validated, for a downstream application, including providing the knowledge graph to an audio generation model as a prompt to generate audio data based on the knowledge graph.