Transferring information across language understanding model domains
Aspects of the present invention provide a technique to validate the transfer of intents or entities between existing natural language model domains (hereafter “domain” or “NLU”) using click logs, a knowledge graph, or both. At least two different types of transfers are possible. Intents from a first domain may be transferred to a second domain. Alternatively or additionally, entities from the second domain may be transferred to an existing intent in the first domain. Either way, additional intent/entity pairs can be generated and validated. Before the new intent/entity pair is added to a domain, aspects of the present invention validate that the intent or entity is transferable between domains. Validation techniques that are consistent with aspects of the invention can use a knowledge graph, search query click logs, or both to validate a transfer of intents or entities from one domain to another.
1. A computer-implemented method of expanding slot coverage for a domain-specific natural language understanding (“NLU”) system, the method comprising:
accessing a plurality of queries from labeled training data for a classifier used to recognize an intent within the domain-specific NLU, wherein the intent is associated with a slot of the domain-specific NLU;
identifying a plurality of entities that occur in the slot of the domain-specific NLU within the plurality of queries;
extracting, from a knowledge graph, a graph type for each of the plurality of entities to generate a plurality of candidate graph types for the slot of the domain-specific NLU, the candidate graph types comprising at least one compatible entity that is eligible for pairing with the recognized intent of the domain-specific NLU;
calculating a correlation score for each graph type in the plurality of candidate graph types for pairing the at least one compatible entity with the recognized intent of the domain-specific NLU;
assigning an individual graph type having the highest correlation score as the graph type the domain-specific NLU slot can accept;
expanding slot coverage for the domain-specific NLU by validating the pairing of the at least one compatible entity associated with the assigned individual graph type with the recognized intent of the domain-specific NLU; and
utilizing the validated pairing of the compatible entity and the recognized intent to interpret a natural language input of a query.
2. The method of claim 1 , wherein the method further comprises associating a new graph type with the domain-specific NLU slot upon determining the new graph type is a child within the knowledge graph of the individual graph type.
3. The method of claim 1 , wherein the method further comprises associating a new graph type with the domain-specific NLU slot upon determining the new graph type has an attribute within the knowledge graph of the individual graph type.
4. The method of claim 1 , wherein the correlation score is calculated using a Term Frequency/ Inverse document frequency (“TF/IDF”) equation that uses graph types as terms and slots as documents.
5. The method of claim 4 , wherein IDF is calculated to avoid down weighting a graph type that appears with many similar slots by determining a similarity between slots.
6. The method of claim 4 , wherein TF is calculated using a function that gives a weight that is inversely proportional to a number of graph types appearing with each entity in the knowledge graph.
7. The method of claim 1 , wherein the correlation score is calculated using a weighted summation.
8. The method of claim 1 , wherein the method further comprises transferring an entity list from a second domain-specific natural language understanding (“NLU”) system upon determining entities in the entity list are associated with a graph type that is compatible with the individual graph type.
9. One or more computer-storage media having computer-executable instructions embodied thereon that when executed by a computing device perform the method of expanding slot coverage for a domain-specific natural language understanding (“NLU”) system, the method comprising:
accessing a plurality of queries from labeled training data for a classifier used to recognize an intent within the domain-specific NLU, wherein the intent is associated with a slot of the domain-specific NLU;
identifying a plurality of entities that occur in the slot of the domain-specific NLU within the plurality of queries;
extracting, from a knowledge graph, a graph type for each of the plurality of entities to generate a plurality of candidate graph types for the slot of the domain-specific NLU, the candidate graph types comprising at least one compatible entity that is eligible for pairing with the recognized intent of the domain-specific NLU;
calculating a correlation score for each graph type in the plurality of candidate graph types for pairing the at least one compatible entity with the recognized intent of the domain-specific NLU;
assigning an individual graph type having the highest correlation score as the graph type the domain-specific NLU slot can accept;
expanding slot coverage for the domain-specific NLU by validating the pairing of the at least one compatible entity associated with the assigned individual graph type with the recognized intent of the domain-specific NLU; and
utilizing the validated pairing of the compatible entity and the recognized intent to interpret a natural language input of a query.
10. The method of claim 9 , the method further comprising associating a new graph type with the domain-specific NLU slot upon determining the new graph type is a child within the knowledge graph of the individual graph type.
11. The method of claim 9 , wherein the method further comprises associating a new graph type with the domain-specific NLU slot upon determining the new graph type has an attribute within the knowledge graph of the individual graph type.
12. The method of claim 9 , wherein the correlation score is calculated using a Term Frequency/Inverse document frequency (“TF/IDF”) equation that uses graph types as terms and slots as documents.
13. The method of claim 12 , wherein IDF is calculated to avoid down weighting a graph type that appears with many similar slots by determining a similarity between slots.
14. The method of claim 12 , wherein TF is calculated using a function that gives a weight that is inversely proportional to a number of graph types appearing with each entity in the knowledge graph.
15. The method of claim 9 , wherein the correlation score is calculated using a weighted summation.
16. The method of claim 9 , wherein the method further comprises transferring an entity list from a second domain-specific natural language understanding (“NLU”) system upon determining entities in the entity list are associated with a graph type that is compatible with the individual graph type.
17. A computer-implemented method of expanding slot coverage for a domain-specific natural language understanding (“NLU”) system, the method comprising:
accessing a plurality of queries from training data to recognize an intent and a plurality of domain-specific entities, wherein the recognized intent and the plurality of domain-specific entities are associated with a slot of the domain-specific NLU;
generating a plurality of candidate graph types for the slot of the domain-specific NLU by extracting, from a knowledge graph, a plurality of candidate graph types associated with each of the plurality of domain-specific entities, the candidate graph types comprising at least one compatible entity that is eligible for pairing with the recognized intent of the domain-specific NLU;
assigning at least one of the plurality of candidate graph types as an individual graph type that the domain-specific NLU can accept based on a correlation score for each of the plurality of candidate graph types associated with the domain-specific entity, the individual graph type having the highest correlation score;
expanding slot coverage of the domain-specific NLU by validating the pairing of the at least one compatible entity associated with the assigned graph type with the recognized intent of the domain-specific NLU; and
utilizing the validated pairing of the compatible entity and the recognized intent to interpret a natural language input of a query.
18. The method of claim 17 , wherein the method further comprises associating a new graph type with the domain-specific NLU slot upon determining the new graph type is a child within the knowledge graph of the individual graph type.
19. The method of claim 17 , wherein the method further comprises associating a new graph type with the domain-specific NLU slot upon determining the new graph type has an attribute within the knowledge graph of the individual graph type.
20. The method of claim 18 , wherein the correlation score is calculated using a weighted summation.
21. The method of claim 18 , wherein the method further comprises transferring an entity list from a second domain-specific natural language understanding (“NLU”) system upon determining entities in the entity list are associated with a graph type that is compatible with the individual graph type.