Systems and methods for factual extraction from language model

1. A method for extracting factual information from a language model, the method comprising:

receiving, via a communication interface, a query for factual information;

encoding, via an embedding layer of a pre-trained language model, the natural language prompt into a first embedding;

encoding, via an adapter model the first embedding into a second embedding that includes a continuous representation based on a probability that the second embedding will return the factual information when the second embedding is fed to a first attention layer of the pre-trained language model, wherein the adapter model is placed between the embedding layer of the pre-trained language model and a first attention layer of the pre-trained language model;

decoding, via the first attention layer of the pre-trained language model, the second embedding into a response to the query; and

extracting the factual information from the decoded response to the query.

2. The method of claim 1 , wherein encoding, via the adapter model, the first embedding into a second embedding comprises;

determining, via an LSTM (Long Short Term Memory) layer, an LSTM encoding in response to the first embedding; and

determining, via a Multi-Layer Perceptron (MLP), the second embedding in response to the LSTM encoding.

3. The method of claim 1 , wherein encoding, via the adapter model, the first embedding into a second embedding comprises;

determining, via an LSTM (Long Short Term Memory) layer, an LSTM encoding based on the first embedding;

down-sampling the LSTM output, via a pooling layer; and

determining, via a Multi-Layer Perceptron (MLP), a part of the second embedding based on the down-sampled LSTM encoding.

4. The method of claim 3 , further comprising;

transmitting the first embedding as part of the second embedding to the first attention layer.

5. The method of claim 1 , wherein encoding, via the adapter model, the first embedding into a second embedding comprises:

determining a first entity and a second entity in the first embedding;

determining, via a classifier, a relationship between the first entity and the second entity; and

determining, via a look-up table, the second embedding associated with the relationship.

6. The method of claim 1 , wherein encoding, via the adapter model, the first embedding into a second embedding comprises;

determining a first entity and a second entity in the first embedding;

determining, via an oracle model, a relationship between the first entity and the second entity, wherein the oracle model returns a relationship given the first entity and the second entity; and

determining, via a look-up table, the second embedding associated with the relationship.

7. The method of claim 1 , wherein encoding, via the adapter model, the first embedding into a second embedding comprises:

determining, via the adapter model, a plurality of alternate embeddings based on the first embedding;

determining, via the adapter model, a probability associated with each of the plurality of alternate embeddings returning the factual information when fed to the first attention layer of the pre-trained language model; and

determining, via the adapter model, the second embedding based on the probability associated with each of the plurality of alternate embeddings.

8. The method of claim 1 , wherein encoding, via the adapter model, the first embedding into a second embedding comprises:

determining a first entity and a mask in the first embedding;

determining, via the adapter model, a plurality of alternate embeddings based on the position of the first entity and the mask;

determining, via the adapter model, a probability associated with each of the plurality of alternate embeddings returning the factual information when fed to the first attention layer of the pre-trained language model; and

determining, via the adapter model, the second embedding based on the probability associated with each of the plurality of alternate embeddings.

9. The method of claim 1 , further comprising:

receiving, via a communication interface, a training data set, the training dataset including a plurality of sets of similar natural language queries for factual information;

determining a first entity and a second entity in each query in a set of similar natural language queries from the plurality of sets of similar natural language queries;

masking at least the first entity or the second entity in each query in the set of similar natural language queries;

determining a masked embedding for each masked query in the set of similar natural language queries;

determining a response to each masked embedding via the pre-trained language model, via the first attention layer of the pre-trained language model;

determining whether the response matches the masked entity in the natural language query;

determine a probability that a masked embedding will return the response that matches the masked entity in the natural language query; and

update an adapter model that chooses a query where the response from the pre-defined language model matches the masked entity based on the determined probability.

10. A system for extracting factual information from a language model, the system comprising:

a communication interface for receiving a query for factual information;

a memory storing a plurality of machine readable instructions; and

a processor reading and executing the instructions from the memory to perform operations comprising:

receive, via the communication interface, a query for factual information;

transform the query into a natural language prompt in a format of a query subject and a queried relation;

encode, via an embedding layer of a pre-trained language model, the natural language prompt into a first embedding;

encode, via the adapter model, the first embedding into a second embedding based on a probability that the second embedding returns the factual information when the second embedding is fed the first attention layer of the pre-trained language model;

decode, by the first attention layer of the pre-trained language mode, the second embedding into a response to the query; and

extract the factual information from the decoded response to the query.

11. The system of claim 10 , wherein the operation of encoding, via the adapter model, the first embedding into a second embedding comprises:

determine, via an LSTM (Long Short Term Memory) layer, an LSTM encoding based on the first embedding; and

determine, via a Multi-Layer Perceptron (MLP), the second embedding based on the LSTM encoding.

12. The system of claim 10 , wherein the operation of encoding, via the adapter model, the first embedding into a second embedding comprises:

determine, via an LSTM layer, an LSTM encoding based on the first embedding;

down-sample the LSTM output, via a pooling layer; and

determine, via a Multi-Layer Perceptron (MLP), a part of the second embedding based on the down-sampled LSTM encoding.

13. The system of claim 12 , wherein the operation of encoding, via the adapter model, the first embedding into a second embedding comprises:

transmit the first embedding as part of the second embedding to the first attention layer.

14. The system of claim 10 , wherein the operation of encoding, via the adapter model, the first embedding into a second embedding comprises:

determine a first entity and a second entity in the first embedding;

determine, via a classifier, a relationship between the first entity and the second; and

determine, via a look-up table, the second embedding associated with the relationship.

15. The system of claim 10 , wherein the operation of encoding, via the adapter model, the first embedding into a second embedding comprises:

determine a first entity and a second entity in the first embedding;

determine, via an oracle model, a relationship between the first entity and the second entity, wherein the oracle model returns a relationship given the first entity and the second entity; and

determine, via a look-up table, the second embedding associated with the relationship.

16. The system of claim 10 , wherein the operation of encoding, via the adapter model, the first embedding into a second embedding comprises: determining, via the adapter model, a plurality of alternate embeddings based on the first embedding;

determine, via the adapter model, a probability associated with each of the plurality of alternate embeddings returning the factual information when fed to the first attention layer of the pre-trained language model; and

determine, via the adapter model, the second embedding based on the probability associated with each of the plurality of alternate embeddings.

17. The system of claim 10 , wherein the operation of encoding, via the adapter model, the first embedding into a second embedding comprises:

determine a first entity and a mask in the first embedding;

determine, via the adapter model, a plurality of alternate embeddings based on the position of the first entity and the mask;

determine, via the adapter model, a probability associated with each of the plurality of alternate embeddings returning the factual information when fed to the first attention layer of the pre-trained language model; and

determine, via the adapter model, the second embedding based on the probability associated with each of the plurality of alternate embeddings.

18. The system of claim 10 , wherein the processor performs operations comprising:

receive, via a communication interface, a training data set, the training dataset including a plurality of sets of similar natural language queries for factual information;

determine a first entity and a second entity in each query in a set of similar natural language queries from the plurality of sets of similar natural language queries;

generate a mask for at least the first entity or the second entity in each query in the set of similar natural language queries;

determine a masked embedding for each masked query in the set of similar natural language queries;

determine a response to each masked embedding via the pre-trained language model, via the first attention layer of the pre-trained language model;

determine whether the response matches the masked entity in the natural language query;

determine a probability that a masked embedding will return the response that matches the masked entity in the natural language query; and

update an adapter model that chooses a query where the response from the pre-defined language model matches the masked entity based on the determined probability.

19. A processor-readable non-transitory storage medium storing a plurality of processor-executable instructions for extracting factual information from a language model, the instructions being executed by one or more processors to perform operations comprising:

receiving, via a communication interface, a query for factual information;

transforming the query into a natural language prompt in a format of a query subject and a queried relation;

encoding, via an embedding layer of a pre-trained language model, the natural language prompt into a first embedding;

encoding, via an adapter model, the first embedding into a second embedding based on a probability that the second embedding returns the factual information when the second embedding is fed the first attention layer of the pre-trained language model, wherein the adapter model is placed between the embedding layer of the pre-trained language model and the first embedding layer of the pre-trained language model;

decoding, by the first attention layer of the pre-trained language mode, the second embedding into a response to the query; and

extracting the factual information from the decoded response to the query.

20. A processor-readable non-transitory storage medium storing a plurality of processor-executable instructions for extracting factual information from a language model, of claim 19 , the instructions being executed by one or more processors to perform operations comprising:

determining, via an LSTM (Long Short Term Memory) layer, an LSTM encoding based on the first embedding; and

determining, via a Multi-Layer Perceptron (MLP), the second embedding based on the LSTM encoding.