Direct preference optimization data filtering via answer similarity using language model embeddings
Disclosed herein are system, method, apparatus, and computer-readable medium embodiments for filtering data for direct preference optimization training via answer similarity using language model embeddings. A system inputs an accepted question-response pair and a rejected question-response pair into a language model. The system extracts a first embedding for the accepted question-response pair and a second embedding for the rejected question-response pair from the language model. Using the first embedding and the second embedding, the system further computes a similarity score for the accepted question-response pair and the rejected question-response pair. Finally, based on the similarity score satisfying a threshold, the system includes the rejected question-response pair in a dataset and trains the language model using the dataset.
1 . A computer-implemented method, comprising:
inputting an accepted question-response pair and a rejected question-response pair into a language model;
extracting, from the language model, a first embedding for the accepted question-response pair comprising a first set of vectors, wherein a vector of the first set of vectors further comprises at least one numeric value representing a semantic relationship between a token in a second set of one or more tokens associated with an accepted response and a token in a first set of one or more tokens associated with a question;
extracting, from the language model, a second embedding for the rejected question-response pair comprising a second set of vectors, wherein a vector of the second set of vectors further comprises at least one numeric value representing semantic relationship between a token in a third set of one or more tokens associated with a rejected response and the token in the first set of one or more tokens associated with the question;
computing, using the first embedding and the second embedding, a similarity score indicating a similarity between the accepted question-response pair and the rejected question-response pair;
selecting, for inclusion in a dataset, the rejected question-response pair based on the similarity score satisfying a threshold, wherein the dataset comprises a plurality of rejected question-response pairs having a minimum level of similarity with a plurality of corresponding accepted question-response pairs; and
finetuning, using the dataset, the language model.
2 . The computer-implemented method of claim 1 , wherein the extracting further comprises extracting the first embedding and the second embedding from a last hidden layer of the language model.
3 . The computer-implemented method of claim 1 , further comprising:
receiving, from one or more data sources, a first text string specifying the question, a second text string specifying the accepted response, and a third text string specifying the rejected response, wherein the accepted question-response pair and the rejected question-response pair are based on one or more of the first text string, the second text string, or the third text string.
4 . The computer-implemented method of claim 3 , further comprising:
tokenizing the first text string into the first set of one or more tokens;
tokenizing the second text string into the second set of one or more tokens; and
tokenizing the third text string into the third set of one or more tokens.
5 . The computer-implemented method of claim 4 , further comprising:
constructing the accepted question-response pair using the first set of one or more tokens and the second set of one or more tokens; and
constructing the rejected question-response pair using the first set of one or more tokens and the third set of one or more tokens.
6 . The computer-implemented method of claim 1 , wherein computing the similarity score for the accepted question-response pair and the rejected question-response pair comprises one or more of:
(i) computing a cosine similarity between the first embedding and the second embedding;
(ii) computing Euclidean distance between the first embedding and the second embedding; or
(iii) computing n-gram parameters for the first embedding and the second embedding.
7 . The computer-implemented method of claim 1 , wherein computing the similarity score for the accepted question-response pair and the rejected question-response pair comprises applying a second language model to the first embedding and the second embedding.
8 . The computer-implemented method of claim 1 , further comprising:
determining that a second similarity score for the accepted question-response pair and a second rejected question-response pair fails to satisfy the threshold; and
excluding the second rejected question-response pair from the dataset.
9 . The computer-implemented method of claim 1 , wherein training the language model using the dataset comprises performing direct preference optimization.
10 . The computer-implemented method of claim 9 , wherein the language model is an unsupervised language model and performing the direct preference optimization comprises adjusting one or more model parameters based on a cross-entropy loss function and a reference policy that indicates a baseline model behavior.
11 . The computer-implemented method of claim 1 , wherein extracting the first embedding and the second embedding comprises extracting the first embedding and the second embedding without obtaining a final output of the language model regarding the accepted question-response pair or the rejected question-response pair.
12 . The computer-implemented method of claim 1 , wherein the accepted response comprises a preferred content item selected from a set of one or more content items generated by the language model according to a criterion comprising at least one of relevance, completeness, or coherence, and wherein the rejected response comprises a non-preferred content item remaining in the set of one or more content items.
13 . An apparatus comprising a processing system that includes one or more processors and one or more memories coupled with the one or more processors, the processing system configured to:
input an accepted question-response pair and a rejected question-response pair into a language model;
extract, from the language model, a first embedding for the accepted question-response pair comprising a first set of vectors, wherein a vector of the first set of vectors further comprises at least one numeric value representing a semantic relationship between a token in a second set of one or more tokens associated with an accepted response and a token in a first set of one or more tokens associated with a question;
extract, from the language model, a second embedding for the rejected question-response pair comprising a second set of vectors, wherein a vector of the second set of vectors further comprises at least one numeric value representing a semantic relationship between a token in a third set of one or more tokens associated with a rejected response and the token in the first set of one or more tokens associated with the question;
compute, using the first embedding and the second embedding, a similarity score indicating a similarity between the accepted question-response pair and the rejected question-response pair;
select, for inclusion in a dataset, the rejected question-response pair based on the similarity score satisfying a threshold, wherein the dataset comprises a plurality of rejected question-response pairs having a minimum level of similarity with a plurality of corresponding accepted question-response pairs; and
finetuning, using the dataset, the language model.
14 . The apparatus of claim 13 , wherein the processing system is further configured to receive, from one or more data sources, a first text string specifying the question, a second text string specifying the accepted response, and a third text string specifying the rejected response, wherein the accepted question-response pair and the rejected question-response pair are based on one or more of the first text string, the second text string, or the third text string.
15 . The apparatus of claim 14 , wherein the processing system is further configured to:
tokenize the first text string into the first set of one or more tokens;
tokenize the second text string into the second set of one or more tokens; and
tokenize the third text string into the third set of one or more tokens.
16 . The apparatus of claim 15 , wherein the processing system is further configured to:
construct the accepted question-response pair using the first set of one or more tokens and the second set of one or more tokens; and
construct the rejected question-response pair using the first set of one or more tokens and the third set of one or more tokens.
17 . The apparatus of claim 13 , wherein the accepted response comprises a preferred content item selected from a set of one or more content items generated by the language model according to a criterion comprising at least one of relevance, completeness, or coherence, and wherein the rejected response comprises a non-preferred content item remaining in the set of one or more content items.
18 . The apparatus of claim 13 , wherein the processing system is further configured to compute the similarity score for the accepted question-response pair and the rejected question-response pair by performing one or more of:
(i) computing a cosine similarity between the first embedding and the second embedding;
(ii) computing Euclidean distance between the first embedding and the second embedding; or
(iii) computing n-gram parameters for the first embedding and the second embedding.
19 . A computer-implemented method, comprising:
inputting an accepted question-response pair and a rejected question-response pair into a language model;
extracting, from the language model, a first embedding for the accepted question-response pair comprising a first set of vectors, wherein a vector of the first set of vectors further comprises at least one numeric value representing a semantic relationship between a token in a second set of one or more tokens associated with an accepted response and a token in a first set of one or more tokens associated with a question;
extracting, from the language model a second embedding for the rejected question-response pair comprising a second set of vectors, wherein a vector of the second set of vectors further comprises at least one numeric value representing a semantic relationship between a token in a third set of one or more tokens associated with a rejected response and the token in the first set of one or more tokens associated with the question;
computing, using the first embedding and the second embedding, a similarity score indicating a similarity between the accepted question-response pair and the rejected question-response pair;
determining the similarity score satisfies a threshold, wherein the similarity score satisfying the threshold indicates that the rejected question-response pair is associated with at least a minimum level of similarity to the accepted question-response pair; and
updating, by applying direct preference optimization based on the rejected question-response pair, the language model.
20 . The computer-implemented method of claim 19 , wherein the accepted response comprises a preferred content item selected from a set of one or more content items generated by the language model according to a criterion comprising at least one of relevance, completeness, or coherence, and wherein the rejected response comprises a non-preferred content item remaining in the set of one or more content items.