Granted Patent B2

US 12,265,564 · App. 18/146,765 · Granted Apr 1, 2025

Inventors: Hongming Zhang (Palo Alto, CA); Xiaoman Pan (Palo Alto, CA); Wenlin Yao (Mill Creek, WA); Jianshu Chen (Palo Alto, CA); Dong Yu (Palo Alto, CA)

Assignee: TENCENT AMERICA LLC

G06F16/3344G06F16/355

Abstract

Claims (45)

1. A method of instance-wise adaptive knowledge injection in a large language pre-trained language model (PTLM), the method being executed by at least one processor, the method comprising:

determining whether external knowledge is needed for a respective query in a plurality of queries of a first dataset based on a thrust score of the respective query using internal knowledge of a target large scale pre-trained language model, wherein determining the thrust score comprises:

generating a query distribution based on the target lar e scale pre-trained language model;

generating one or more clusters based on the query distribution;

for the respective query among the plurality of queries, determining one or more unit vectors associated with the query that pointing from a query vector of the query to a center of each cluster among the one or more clusters, wherein each unit vector is associated with the query and a respective cluster among the one or more clusters; and

determining the thrust score for the respective query based on a sum vector of the one or more unit vectors weighted by a size of each of the one or more clusters, and

wherein each query in the or more clusters is represented using last layer hidden states of the target large scale pre-trained language model associated with each query;

based on determining that external knowledge is needed for one or more queries among the plurality of queries of the first dataset, augmenting the one or more queries with respective pieces of external knowledge;

generating a combined dataset based on combining the first dataset and the one or more augmented queries; and

applying the combined dataset to the target large scale pre-trained language model.

2. The method of claim 1 , wherein determining whether external knowledge is needed is based on whether the target large scale pre-trained language model has no relevant knowledge, the target large scale pre-trained language model is not familiar with the respective query, or the target large scale pre-trained language model includes controversial knowledge associated with the respective query.

3. The method of claim 2 , wherein the controversial knowledge associated with the respective query comprises the respective query being associated with different questions or the respective query being associated with different reasoning.

4. The method of claim 1 , wherein the thrust score for the respective query among the plurality of queries is further based on a division with a square of a Euclidean distance between the query vector and a center vector at the center of each cluster.

5. The method of claim 1 , wherein the last layer hidden states of the target large scale pre-trained language model are used to generate the query distribution.

6. The method of claim 1 , wherein one or more last layers of decoders of the target large scale pre-trained language model are used to generate the query distribution.

7. An apparatus for instance-wise adaptive knowledge injection in a pre-trained language model (PTLM), the apparatus comprising:

at least one memory configured to store computer program code;

at least one processor configured to access the computer program code and operate as instructed by the computer program code, the computer program code including:

first determining code configured to cause the at least one processor to determine whether external knowledge is needed for a respective query in a plurality of queries of a first dataset based on a thrust score of the respective query using internal knowledge of a target large scale pre-trained language model,

wherein each query in the one or more clusters is represented using last layer hidden states of the target large scale pre-trained language model associated with each query;

based on determining that external knowledge is needed for one or more queries among the plurality of queries of the first dataset, first augmenting code configured to cause the at least one processor to augment the one or more queries with respective pieces of external knowledge;

first generating code configured to cause the at least one processor to generate a combined dataset based on combining the first dataset and the one or more augmented queries; and

first applying code configured to cause the at least one processor to apply the combined dataset to the target large scale pre-trained language model,

wherein the first determining code comprises:

second generating code configured to cause the at least one processor to generate a query distribution based on the target large scale pre-trained language model;

third generating code configured to cause the at least one processor to generate one or more clusters based on the query distribution;

second determining code configured to cause the at least one processor to determine, for the respective query among the plurality of queries, one or more unit vectors associated with the query that pointing from a query vector of the query to a center of each cluster among the one or more clusters, wherein each unit vector is associated with the query and a respective cluster among the one or more clusters; and

third determining code configured to cause the at least one processor to determine the thrust score for the query based on a sum vector of the one or more unit vectors weighted by a size of each of the one or more clusters.

8. The apparatus of claim 7 , wherein determining whether external knowledge is needed is based on whether the target large scale pre-trained language model has no relevant knowledge, the target large scale pre-trained language model is not familiar with the respective query, or the target large scale pre-trained language model includes controversial knowledge associated with the respective query.

9. The apparatus of claim 8 , wherein the controversial knowledge associated with the respective query comprises the respective query being associated with different questions or the respective query being associated with different reasoning.

10. The apparatus of claim 7 , wherein the thrust score for the respective query among the plurality of queries is further based on a division with a square of a Euclidean distance between the query vector and a center vector at the center of each cluster.

11. The apparatus of claim 7 , wherein the last layer hidden states of the target large scale pre-trained language model are used to generate the query distribution.

12. The apparatus of claim 7 , wherein one or more last layers of decoders of the target large scale pre-trained language model are used to generate the query distribution.

13. A non-transitory computer-readable medium storing computer code that is configured to, when executed by at least one processor, cause the at least one processor to implement instance-wise adaptive knowledge injection in a pre-trained language model (PTLM) that:

determines whether external knowledge is needed for a respective query in a plurality of queries of a first dataset based on a thrust score of the respective query using internal knowledge of a target large scale pre-trained language model, wherein determining the thrust score comprises:

generating a query distribution based on the target large scale pre-trained language model,

generating one or more clusters based on the query distribution,

for the respective query among the plurality of queries, determining one or more unit vectors associated with the query that pointing from a query vector of the query to a center of each cluster among the one or more clusters, wherein each unit vector is associated with the query and a respective cluster among the one or more clusters,

determining the thrust score for the respective query based on a sum vector of the one or more unit vectors weighted by a size of each of the one or more clusters, and

wherein each query in the one or more clusters is represented using last layer hidden states of the target large scale pre-trained language model associated with each query;

based on determining that external knowledge is needed for one or more queries among the plurality of queries of the first dataset, augments the one or more queries with respective pieces of external knowledge;

generates a combined dataset based on combining the first dataset and the one or more augmented queries; and

applies the combined dataset to the target large scale pre-trained language model.

14. The non-transitory computer-readable medium of claim 13 , wherein determining whether external knowledge is needed is based on whether the target large scale pre-trained language model has no relevant knowledge, the target large scale pre-trained language model is not familiar with the respective query, or the target large scale pre-trained language model includes controversial knowledge associated with the respective query.

15. The non-transitory computer-readable medium of claim 14 , wherein the controversial knowledge associated with the respective query comprises the respective query being associated with different questions or the respective query being associated with different reasoning.

Assignments (2)

Continuity (1)

References Cited (13)