Generating commonsense context for text using knowledge graphs
Methods and systems are provided for facilitating generation and utilization of a commonsense contextualizing machine learning (ML) model, in accordance with embodiments described herein. In embodiments, a commonsense contextual ML model is trained by fine-tuning a pre-trained language model using a set of training path-sentence pairs. Each training path-sentence pair includes a commonsense path, identified via a commonsense knowledge graph, and a natural language sentence identified as contextually related to the commonsense path. The trained commonsense contextualizing ML model can then be used to generate a commonsense inference path for a text input. Such a commonsense inference path can include a sequence of entities and relations that provide commonsense context to the text input. Thereafter, the commonsense inference path can be provided to a natural language processing system for use in performing a natural language processing task.
1. A computer-implemented method comprising:
generating a commonsense path derived using a commonsense knowledge graph, the commonsense path including a sequence comprising entities and relations;
generating a training path-sentence pair including the commonsense path from the commonsense knowledge graph and a natural language sentence identified as contextually-related to the commonsense path based on at least one entity or relation associated with the commonsense path, wherein the natural language sentence is identified as contextually-related to the commonsense path using a query including an entity and a relation from the commonsense path generated via a query template to query a set of natural language sentences and obtain a subset of natural language sentences determined to be relevant to the query and using a model that derives semantically meaningful sentence embeddings to identify the natural language sentence, from the subset of natural language sentences, as within a threshold of similarity to the commonsense path;
using the training path-sentence pair to train a commonsense contextualizing machine learning model comprising a pre-trained language model;
using the commonsense contextualizing machine learning model to generate a commonsense inference path for an input text provided to the commonsense contextualizing machine learning model, the commonsense inference path including a sequence comprising entities and relations that provide commonsense context to the input text, the commonsense inference path generated by concatenating a set of path tokens, wherein each path token represents an entity or a relation at a corresponding time-step associated with the commonsense contextualizing machine learning model; and
providing, to a natural language processing system, the input text and the commonsense inference path to augment the input text for use in performing a natural language processing task using at least the commonsense inference path.
2. The computer-implemented method of claim 1 , wherein the commonsense path is identified from the commonsense knowledge graph based on an extraction of a set of multi-hop paths.
3. The computer-implemented method of claim 2 , wherein the length of each path of the set of multi-hop paths is within a predetermined range of hops.
4. The computer-implemented method of claim 1 , further comprising obtaining a set of natural language sentences, extracted from an electronic source of sentences, for use in generating a set of training path-sentence pairs including the training path-sentence pair.
5. The computer-implemented method of claim 1 , wherein a set of query templates are used to generate a set of queries for querying the set of natural language sentences.
6. The computer-implemented method of claim 1 , wherein the natural language sentence is identified as contextually-related to the commonsense path by:
accessing a set of query templates;
using the set of query templates to generate a set of queries based on the commonsense path; and
query an index having the set of natural language sentences to identify the natural language sentence as matching the query of the set of queries.
7. The computer-implemented method of claim 1 , wherein the training path-sentence pair is used to train the commonsense contextualizing machine learning model by:
inputting the training sentence into the pre-trained language model; and
using an output commonsense inference path from the pre-trained model to compare to the commonsense path of the training path-sentence pair to fine tune the commonsense contextualizing machine learning model.
8. The computer-implemented method of claim 1 , wherein training the commonsense contextualizing machine learning model includes performing masking of an entity in the natural language sentence, the entity also being included in the commonsense path.
9. One or more non-transitory computer-readable storage media having a plurality of executable instructions embodied thereon, which, when executed by one or more processors, cause the one or more processors to perform operations comprising:
obtaining an input text;
inputting the input text into a commonsense contextualizing machine learning model trained based on a set of path-sentence pairs, wherein each path-sentence pair includes a commonsense path identified via a commonsense knowledge graph and a natural language sentence identified as contextually-related to the commonsense path using a query generated to query a set of natural language sentences and obtain a subset of natural language sentences and using a model that derives semantically meaningful sentence embeddings to identify the natural language sentence, from the subset of natural language sentences, as within a threshold of similarity to the commonsense path;
obtaining, as output from the commonsense contextualizing model, a commonsense inference path including a set of entities and relations providing commonsense context relevant to the input text, the commonsense inference path generated by concatenating a set of path tokens, wherein each path token represents an entity or a relation at a corresponding time-step associated with the commonsense contextualizing model; and
providing the input text with the commonsense inference path to augment the input text to a natural language processing system for performing a natural language processing task using the commonsense inference path.
10. The media of claim 9 , wherein the commonsense contextualizing machine learning model is trained using a pre-trained natural language model.
11. The media of claim 10 , wherein the commonsense contextualizing machine learning model outputs a set of diverse commonsense inference paths.
12. The media of claim 11 , wherein the set of diverse commonsense inference paths is generated based on sampling a predetermined number of most probable tokens at a first level associated with the commonsense contextualizing machine learning model and selecting a most probable path forward for each of the predetermined number of most probable tokens at the first level.
13. The media of claim 9 , wherein the natural language processing system uses at least the commonsense inference path to perform the natural language processing task.
14. The media of claim 13 , wherein the natural language processing task comprises at least one of a conversational bot task, a dialogue agent task, an information retrieval task, or a question answering task.
15. The media of claim 9 , wherein the natural language sentence is identified as contextually-related to the commonsense path by:
accessing a set of query templates;
using the set of query templates to generate a set of queries based on the commonsense path; and
querying an index having the set of natural language sentences to identify the natural language sentence as matching the query of the set of queries.
16. A system comprising:
a processor device; and
a memory device, coupled with the processor device, the processor device to perform actions comprising:
training a commonsense contextual machine learning (ML) model by fine tuning a pre-trained language model based on a set of training path-sentence pairs, wherein each training path-sentence pair includes a commonsense path, identified via a commonsense knowledge graph and a natural language sentence identified as contextually-related to the commonsense path, wherein the natural language sentence is identified as contextually-related to the commonsense path based on a query, generated using a query template, to search a set of natural language sentences to identify a subset of natural language sentences and using a model that derives semantically meaningful sentence embeddings to identify the natural language sentence, from the subset of natural language sentences, as within a threshold of similarity to the commonsense path;
using the commonsense contextualizing ML model to generate a commonsense inference path for a text input into the commonsense contextualizing ML model, the commonsense inference path including a sequence comprising entities and relations that provide commonsense context to the text input, the commonsense inference path generated by concatenating a set of path tokens, wherein each path token represents an entity or a relation at a corresponding time-step associated with the commonsense contextualizing ML model; and
providing the text input with the commonsense inference path to augment the text input to a natural language processing system for use in performing a natural language processing task using the text and the commonsense inference path.
17. The system of claim 16 , wherein the natural language processing task comprises at least one of a conversational bot task, a dialogue agent task, an information retrieval task, or a question answering task.
18. The system of claim 17 , wherein the sequence included in the commonsense inference path includes a portion selected from the text input and a portion not included within the text input.
19. The system of claim 16 , wherein the commonsense contextualizing ML model outputs a set of diverse commonsense inference paths that are provided to the natural language processing system for use in performing the natural language processing task.