Language model with external knowledge base
The technology described herein receives a natural-language sequence of words comprising multiple entities. The technology then identifies a plurality of entities in the natural-language sequence. The technology generates a masked natural-language sequence by masking a first entity in the natural-language sequence. The technology retrieves, from a knowledge base, information related to a second entity in the plurality of entities. The technology then trains a natural-language model to respond to a query. The training uses a first representation of the masked natural-language sequence, a second representation of the information, and the first entity.
1. A computer-implemented method comprising:
receiving a natural-language sequence of words comprising multiple entities, wherein the entities comprise persons and places;
identifying a plurality of entities in the natural-language sequence;
generating a masked natural-language sequence by masking a first entity in the natural-language sequence;
generating a first representation of the masked natural-language sequence, wherein the first representation is a machine embedding of the masked natural-language sequence;
retrieving, from a knowledge base, information related to a second entity in the plurality of entities;
generating a second representation of the information; and
training a natural-language model to respond to a query, wherein the training uses the first representation of the masked natural-language sequence and the second representation of the information as inputs and the first entity as a training label.
2. The computer-implemented method of claim 1 , wherein the information is a triple comprising the second entity, a third entity, and a relationship between the second entity and the third entity.
3. The computer-implemented method of claim 2 , further comprising generating a natural language phrase that includes the second entity, the third entity and the relationship.
4. The computer-implemented method of claim 3 , wherein the second representation of the information is a machine embedding of the natural language phrase.
5. The computer-implemented method of claim 2 , further comprising generating a similarity score between the first representation of the natural-language sequence of words and the second representation of the information.
6. The computer-implemented method of claim 5 , wherein the similarity score is based on a relational similarity score between a machine embedding of the relationship from the triple and a machine embedding of the natural-language sequence of words.
7. The computer-implemented method of claim 5 , wherein the information is associated with the similarity score above a threshold rank when stack ranked with similarity scores calculated for other information retrieved from the knowledge base.
8. The computer-implemented method of claim 1 , wherein the training comprises masked language modelling where a training objective is to predict the first entity.
9. A non-transitory computer-readable medium storing computer-executable instructions that, when executed by a processing device, cause the processing device to:
receiving a query comprising a natural-language sequence of words;
identifying a plurality of entities in the natural-language sequence of words, wherein the entities comprise persons and places;
retrieving, from a knowledge base, information related to an entity in the plurality of entities, wherein the information is a triple comprising the entity, a second entity, and a relationship between the entity and the second entity;
generating a natural language phrase that includes the entity, the second entity and the relationship;
generating a first representation of the natural-language sequence comprising a machine embedding of the natural-language sequence;
generating a second representation of the information comprising a machine embedding of the natural language phrase;
providing the first representation of the natural-language sequence of words and the second representation of the information to a natural-language model;
in response to the providing, generating, using the natural-language model, a natural language response to the query; and
communicating the natural language response.
10. The non-transitory computer-readable medium of claim 9 , wherein the natural language response is provided by a chat bot.
11. The non-transitory computer-readable medium of claim 9 , wherein the response includes a natural language phrase.
12. The non-transitory computer-readable medium of claim 9 , further comprising generating a similarity score between the first representation of the natural-language sequence of words and the second representation of the information.
13. The non-transitory computer-readable medium of claim 12 , wherein the similarity score is based on a relational similarity score between a machine embedding of the relationship from the triple and the machine embedding of the natural-language sequence of words.
14. The non-transitory computer-readable medium of claim 12 , wherein the information is associated with the similarity score above a threshold rank when stack ranked with similarity scores calculated for other information retrieved from the knowledge base.
15. A system comprising:
a memory component; and
a processing device coupled to the memory component, the processing device to perform operations comprising:
receiving a natural-language sequence of words;
identifying an entity in the natural-language sequence of words;
retrieving, from a knowledge base, a plurality of triples related to the entity, wherein each triple in plurality of triples comprises the entity, a second entity, and a relationship between the entity and the second entity;
for each triple in the plurality of triples, calculating a similarity score that represents an amount of similarity between the natural-language sequence of words and an individual triple;
selecting a top plurality of triples from the plurality of triples using the similarity score;
providing a first representation of the natural-language sequence of words and a second representation of the top plurality of triples to a natural-language model;
in response to the providing, generating, using the natural-language model, a natural language response to the natural-language sequence of words; and
communicating the natural language response to a user.
16. The system of claim 15 , wherein each of the plurality of triples comprise the entity, a second entity, and a relationship between the entity and the second entity.
17. The system of claim 16 , further comprising generating a natural language phrase that includes the entity, the second entity and the relationship.
18. The system of claim 17 , wherein the first representation of the natural-language sequence is a machine embedding of the natural-language sequence and the second representation of the top plurality of triples includes a machine embedding of the natural language phrase.
19. The system of claim 18 , wherein the similarity score is based on a relational similarity score between a machine embedding of the relationship from the individual triple and a machine embedding of the natural-language sequence of words.
20. The system of claim 18 , wherein the natural language response is provided by a chat bot.