Tied Preference Optimization for Sequence Processing Models
Provided are systems and methods for fine-tuning sequence processing models to human preferences. The approaches can account for tied preferences between pairs of sequences and, therefore, can be referred to as Tied Preference Optimization (TPO). Example sequence processing models include so-called large language models (LLMs), large multimodal models (LMMs), and other models that are configured to process inputs and/or generate outputs that are structured as a series of data elements such as tokens.
1 . A method for generating a preference optimization loss function, the method comprising:
selecting a reward function, a regularizer, and a pairwise loss function;
optimizing a single-sequence generation objective which maximizes the selected reward function with regularization according to the selected regularizer, wherein said optimizing results in a solution expression for the selected reward function, wherein the solution expression is a function of a target probability associated with a target sequence processing model and a reference probability associated with a reference sequence processing model;
expressing a pairwise reward difference between the solution expression applied to a first sequence of tokens and the solution expression applied to a second sequence of tokens; and
generating the preference optimization loss function by applying the selected pairwise loss function to fit the pairwise reward difference to a human preference label, wherein the human pairwise preference label comprises a single label that describes a preference between the first sequence of tokens and the second sequence of tokens.
2 . The method of claim 1 , wherein optimizing the single-sequence generation objective comprises distributing respective probabilities for a first tied preference event and a second tied preference event to label values for a first non-tied preference event and a second non-tied preference event.
3 . The method of claim 1 , wherein the selected reward function comprises a per-sequence expected probability of a positive preference.
4 . The method of claim 1 , wherein the selected reward function comprises a logit score or a log of a preference probability.
5 . The method of claim 4 , wherein the selected pairwise loss function comprises a median loss function.
6 . The method of claim 1 , wherein the selected reward function comprises a logit score, wherein the selected pairwise loss function comprises a square loss function, and wherein the human pairwise preference label comprises a fractional preference label.
7 . The method of claim 1 , wherein the regularizer comprises a distance measure applied between a target distribution of the target sequence processing model and a reference distribution of the reference sequence processing model.
8 . The method of claim 7 , wherein the selected regularizer comprises a reverse KL divergence between the reference distribution and the target distribution.
9 . The method of claim 1 , wherein the selected reward function comprises a preference probability score and the selected pairwise loss function comprises a cross entropy loss.
10 . The method of claim 1 , wherein the selected reward function comprises a probability reward function, and wherein the selected regularizer comprises: a combination of a KL divergence and a reverse KL divergence; a Jensen-Shannon divergence; or an Lp regularizer, excluding L zero and L infinity.
11 . The method of claim 1 , wherein the selected pairwise loss function comprises: a cross entropy loss, a square loss, a median loss, or a hinge loss.
12 . A computing system for preference optimization of sequence processing models, the computing system comprising one or more processors and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining, by the computing system, a pairwise preference training example comprising a first sequence of tokens, a second sequence of tokens, and a pairwise preference label, wherein the pairwise preference label comprises one or more label values corresponding to a first non-tied preference event in which the first sequence of tokens is preferred over the second sequence of tokens or a second non-tied preference event in which the second sequence of tokens is preferred over the first sequence of tokens;
evaluating, by the computing system, a tied preference optimization loss function that comprises a pairwise preference probability expression that represents a predicted likelihood of the first non-tied preference event, wherein the pairwise preference probability expression results from distributing respective probabilities for a first tied preference event and a second tied preference event to the label values for the first non-tied preference event and the second non-tied preference event, wherein the first tied preference event comprises neither the first sequence of tokens nor the second sequence of tokens being preferred and the second tied preference event comprises both the first sequence of tokens and the second sequence of tokens being preferred; and
modifying, by the computing system, one or more values of one or more parameters of a target sequence processing model based on the tied preference optimization loss function.
13 . The computing system of claim 12 , wherein the pairwise preference probability expression represents the predicted likelihood of the first non-tied preference event based on first and second target probabilities respectively generated by the target sequence processing model for the first and second sequences of tokens and first and second reference probabilities respectively generated by a reference sequence processing model for the first and second sequences of tokens.
14 . The computing system of claim 13 , wherein the tied preference optimization loss function comprises a negative expectation of a first logarithm of a first expression, the first expression comprising one plus a hyperparameter times a second logarithm of a first ratio of the first target probability to the first reference probability minus the hyperparameter times a third logarithm of a second ratio of the second target probability to the second reference probability.
15 . The computing system of claim 14 , wherein evaluating the tied preference optimization loss function comprises clipping the first expression within the first logarithm to enforce constraints on a difference between first and second positive preference probabilities for the first and second sequences of tokens.
16 . The computing system of claim 13 , wherein the tied preference optimization loss function comprises an expectation of an absolute value of a first expression, the first expression comprising a hyperparameter times a first logarithm of a first ratio of the first target probability to the first reference probability minus the hyperparameter times a second logarithm of a second ratio of the second target probability to the second reference probability minus one.
17 . The computing system of claim 13 , wherein the tied preference optimization loss function comprises an expectation of an absolute value of a first expression, the first expression comprising the minimum between zero or a second expression, the second expression comprising a hyperparameter times a first logarithm of a first ratio of the first target probability to the first reference probability minus the hyperparameter times a second logarithm of a second ratio of the second target probability to the second reference probability minus one.
18 . One or more non-transitory computer-readable media that collectively store a target sequence processing model that has been trained using a preference optimization loss function, the preference optimization loss function having been generated through performance of operations, the operations comprising:
obtaining a general objective comprising a reward function of a learned per-sequence expected probability of a positive preference and a distance measure applied between a target distribution of the target sequence processing model and a reference distribution of a reference sequence processing model;
solving the general objective for a solution expression of the target distribution of the target sequence processing model for a particular output sequence of tokens;
expressing, in terms of the solution expression, a difference between the reward function applied to a first sequence of tokens and the reward function applied to a second sequence of tokens; and
applying a pairwise loss to match the difference to a pairwise preference label associated with the first sequence of tokens and the second sequence of tokens.
19 . The one or more non-transitory computer-readable media of claim 18 , wherein the reward function comprises a preference probability score and the pairwise loss comprises a cross entropy loss.
20 . The one or more non-transitory computer-readable media of claim 18 , wherein the distance measure comprises a reverse KL divergence between the reference distribution and the target distribution.
21 . The one or more non-transitory computer-readable media of claim 18 , wherein the distance measure comprises a combination of a KL divergence and a reverse KL divergence.
22 . The one or more non-transitory computer-readable media of claim 18 , wherein the distance measure comprises a Jensen-Shannon divergence.
23 . The one or more non-transitory computer-readable media of claim 18 , wherein the distance measure comprises an Lp regularizer, excluding L zero and L infinity.
24 . The one or more non-transitory computer-readable media of claim 18 , wherein the pairwise loss comprises: a cross entropy loss, a square loss, or a hinge loss.
25 . The one or more non-transitory computer-readable media of claim 18 , wherein the preference optimization loss function evaluates first and second target probabilities respectively generated by the target sequence processing model for the first and second sequences of tokens and first and second reference probabilities respectively generated by the reference sequence processing model for the first and second sequences of tokens.
26 . The one or more non-transitory computer-readable media of claim 25 , wherein the preference optimization loss function comprises a negative expectation of a logarithm of a sigmoid of a first expression, the first expression comprising a hyperparameter times a first ratio of the second reference probability to the second target probability minus the hyperparameter times a second ratio of the first reference probability to the first target probability, and wherein the expectation is conditioned on the first sequence being preferred over the second sequence.
27 . The one or more non-transitory computer-readable media of claim 25 , wherein the preference optimization loss function comprises an expectation of a square of a first expression, the first expression comprising a hyperparameter times a first ratio of the second reference probability to the second target probability minus the hyperparameter times a second ratio of the first reference probability to the first target probability minus one.
28 . The one or more non-transitory computer-readable media of claim 25 , wherein the preference optimization loss function comprises an expectation of a square of a first expression, the first expression comprising a hyperparameter divided by two times a first logarithm of one plus a first ratio of the second reference probability to the second target probability minus the hyperparameter divided by two times a second logarithm of one plus a second ratio of the first reference probability to the first target probability minus one.
29 . The one or more non-transitory computer-readable media of claim 25 , wherein the preference optimization loss function comprises an expectation of a square of a first expression, the first expression comprising a hyperparameter times the first target probability minus the first reference probability minus the second target probability plus the second reference probability minus one-half or a fractional preference label.