Direct preference optimization data filtering via answer similarity using language model embeddings

1 . A computer-implemented method, comprising:

inputting an accepted question-response pair and a rejected question-response pair into a language model;

extracting, from the language model, a first embedding for the accepted question-response pair comprising a first set of vectors, wherein a vector of the first set of vectors further comprises at least one numeric value representing a semantic relationship between a token in a second set of one or more tokens associated with an accepted response and a token in a first set of one or more tokens associated with a question;

extracting, from the language model, a second embedding for the rejected question-response pair comprising a second set of vectors, wherein a vector of the second set of vectors further comprises at least one numeric value representing semantic relationship between a token in a third set of one or more tokens associated with a rejected response and the token in the first set of one or more tokens associated with the question;

computing, using the first embedding and the second embedding, a similarity score indicating a similarity between the accepted question-response pair and the rejected question-response pair;

selecting, for inclusion in a dataset, the rejected question-response pair based on the similarity score satisfying a threshold, wherein the dataset comprises a plurality of rejected question-response pairs having a minimum level of similarity with a plurality of corresponding accepted question-response pairs; and

finetuning, using the dataset, the language model.

2 . The computer-implemented method of claim 1 , wherein the extracting further comprises extracting the first embedding and the second embedding from a last hidden layer of the language model.

3 . The computer-implemented method of claim 1 , further comprising:

receiving, from one or more data sources, a first text string specifying the question, a second text string specifying the accepted response, and a third text string specifying the rejected response, wherein the accepted question-response pair and the rejected question-response pair are based on one or more of the first text string, the second text string, or the third text string.

4 . The computer-implemented method of claim 3 , further comprising:

tokenizing the first text string into the first set of one or more tokens;

tokenizing the second text string into the second set of one or more tokens; and

tokenizing the third text string into the third set of one or more tokens.

5 . The computer-implemented method of claim 4 , further comprising:

constructing the accepted question-response pair using the first set of one or more tokens and the second set of one or more tokens; and

constructing the rejected question-response pair using the first set of one or more tokens and the third set of one or more tokens.

6 . The computer-implemented method of claim 1 , wherein computing the similarity score for the accepted question-response pair and the rejected question-response pair comprises one or more of:

(i) computing a cosine similarity between the first embedding and the second embedding;

(ii) computing Euclidean distance between the first embedding and the second embedding; or

(iii) computing n-gram parameters for the first embedding and the second embedding.

7 . The computer-implemented method of claim 1 , wherein computing the similarity score for the accepted question-response pair and the rejected question-response pair comprises applying a second language model to the first embedding and the second embedding.

8 . The computer-implemented method of claim 1 , further comprising:

determining that a second similarity score for the accepted question-response pair and a second rejected question-response pair fails to satisfy the threshold; and

excluding the second rejected question-response pair from the dataset.

9 . The computer-implemented method of claim 1 , wherein training the language model using the dataset comprises performing direct preference optimization.

10 . The computer-implemented method of claim 9 , wherein the language model is an unsupervised language model and performing the direct preference optimization comprises adjusting one or more model parameters based on a cross-entropy loss function and a reference policy that indicates a baseline model behavior.

11 . The computer-implemented method of claim 1 , wherein extracting the first embedding and the second embedding comprises extracting the first embedding and the second embedding without obtaining a final output of the language model regarding the accepted question-response pair or the rejected question-response pair.

12 . The computer-implemented method of claim 1 , wherein the accepted response comprises a preferred content item selected from a set of one or more content items generated by the language model according to a criterion comprising at least one of relevance, completeness, or coherence, and wherein the rejected response comprises a non-preferred content item remaining in the set of one or more content items.

13 . An apparatus comprising a processing system that includes one or more processors and one or more memories coupled with the one or more processors, the processing system configured to:

input an accepted question-response pair and a rejected question-response pair into a language model;

extract, from the language model, a first embedding for the accepted question-response pair comprising a first set of vectors, wherein a vector of the first set of vectors further comprises at least one numeric value representing a semantic relationship between a token in a second set of one or more tokens associated with an accepted response and a token in a first set of one or more tokens associated with a question;

extract, from the language model, a second embedding for the rejected question-response pair comprising a second set of vectors, wherein a vector of the second set of vectors further comprises at least one numeric value representing a semantic relationship between a token in a third set of one or more tokens associated with a rejected response and the token in the first set of one or more tokens associated with the question;

compute, using the first embedding and the second embedding, a similarity score indicating a similarity between the accepted question-response pair and the rejected question-response pair;

select, for inclusion in a dataset, the rejected question-response pair based on the similarity score satisfying a threshold, wherein the dataset comprises a plurality of rejected question-response pairs having a minimum level of similarity with a plurality of corresponding accepted question-response pairs; and

finetuning, using the dataset, the language model.

14 . The apparatus of claim 13 , wherein the processing system is further configured to receive, from one or more data sources, a first text string specifying the question, a second text string specifying the accepted response, and a third text string specifying the rejected response, wherein the accepted question-response pair and the rejected question-response pair are based on one or more of the first text string, the second text string, or the third text string.

15 . The apparatus of claim 14 , wherein the processing system is further configured to:

tokenize the first text string into the first set of one or more tokens;

tokenize the second text string into the second set of one or more tokens; and

tokenize the third text string into the third set of one or more tokens.

16 . The apparatus of claim 15 , wherein the processing system is further configured to:

construct the accepted question-response pair using the first set of one or more tokens and the second set of one or more tokens; and

construct the rejected question-response pair using the first set of one or more tokens and the third set of one or more tokens.

17 . The apparatus of claim 13 , wherein the accepted response comprises a preferred content item selected from a set of one or more content items generated by the language model according to a criterion comprising at least one of relevance, completeness, or coherence, and wherein the rejected response comprises a non-preferred content item remaining in the set of one or more content items.

18 . The apparatus of claim 13 , wherein the processing system is further configured to compute the similarity score for the accepted question-response pair and the rejected question-response pair by performing one or more of:

(i) computing a cosine similarity between the first embedding and the second embedding;

(ii) computing Euclidean distance between the first embedding and the second embedding; or

(iii) computing n-gram parameters for the first embedding and the second embedding.

19 . A computer-implemented method, comprising:

inputting an accepted question-response pair and a rejected question-response pair into a language model;

extracting, from the language model, a first embedding for the accepted question-response pair comprising a first set of vectors, wherein a vector of the first set of vectors further comprises at least one numeric value representing a semantic relationship between a token in a second set of one or more tokens associated with an accepted response and a token in a first set of one or more tokens associated with a question;

extracting, from the language model a second embedding for the rejected question-response pair comprising a second set of vectors, wherein a vector of the second set of vectors further comprises at least one numeric value representing a semantic relationship between a token in a third set of one or more tokens associated with a rejected response and the token in the first set of one or more tokens associated with the question;

computing, using the first embedding and the second embedding, a similarity score indicating a similarity between the accepted question-response pair and the rejected question-response pair;

determining the similarity score satisfies a threshold, wherein the similarity score satisfying the threshold indicates that the rejected question-response pair is associated with at least a minimum level of similarity to the accepted question-response pair; and

updating, by applying direct preference optimization based on the rejected question-response pair, the language model.

20 . The computer-implemented method of claim 19 , wherein the accepted response comprises a preferred content item selected from a set of one or more content items generated by the language model according to a criterion comprising at least one of relevance, completeness, or coherence, and wherein the rejected response comprises a non-preferred content item remaining in the set of one or more content items.