Language model training for direct preference optimization and improved language model

1 . A method for training a language model, the method comprising:

executing, a plurality of times, a training step, wherein the training step comprises:

executing, on a prompt, a preferred output, and a non-preferred output, a reference language model to generate a reference score;

executing, on the prompt, the preferred output, and the non-preferred output, a training language model to generate a policy score;

executing a loss function, wherein:

the loss function comprises a combination of the policy score and the reference score,

the loss function further comprises a hyperparameter that modifies the combination of the policy score and the reference score,

the hyperparameter comprises a variable term, α, wherein α varies with a number of training steps performed, including the training step, and

executing the loss function outputs a loss;

generating, from the loss, an updated parameter;

updating the training language model by adjusting an initial parameter of the training language model to the updated parameter, wherein updating generates an updated language model; and

incrementing the number of trainings steps;

determining, after executing the training step the plurality of times, that convergence has occurred; and

returning, after convergence, the updated language model as an improved language model.

2 . The method of claim 1 , wherein the training step improves an alignment of the improved language model to the preferred output over the non-preferred output, relative to the training language model.

3 . The method of claim 1 , wherein the training step increases a stability of the improved language model relative to the training language model.

4 . The method of claim 1 , wherein the training language model comprises a neural network having a plurality of training parameters connecting layers of nodes in the neural network and the reference language model comprises the neural network having a plurality of reference parameters connecting layers of nodes in the neural network.

5 . The method of claim 4 , wherein the training parameters are permitted to change during training and the reference parameters are unchangeable during training.

6 . The method of claim 4 , wherein the training language model outputs a training chosen score and a training reject score, and the reference language model outputs a reference chosen score and a reference reject score.

7 . The method of claim 6 , wherein the training step further comprises:

determining, from the training chosen score and the training reject score, the policy score; and

determining, from the reference chosen score and the reference reject score, the reference score.

8 . The method of claim 7 , wherein the loss function is based on the policy score, the reference score, and the hyperparameter.

9 . The method of claim 1 , wherein the variable term, α, comprises a discontinuous function of the number of training steps.

10 . The method of claim 1 , wherein the variable term, α, comprises a continuous second order function of the number of training steps.

11 . The method of claim 1 , wherein the variable term, α, comprises an increasing phase, a stable phase, and a decreasing phase.

12 . The method of claim 11 , wherein:

during the increasing phase, a value of the variable term, α, increases with the number of training steps,

during the stable phase, the value of the variable term, α, remains unchanged with the number of training steps, and

during the decreasing phase, the value of the variable term, α, decreases with the number of training steps.

13 . The method of claim 11 , wherein:

during the increasing phase, a value of the variable term, α, increases with the number of training steps according to a constant combined with a proportion of a current number of training steps to a first predetermined number of training steps defining the increasing phase,

during the stable phase, the value of the variable term, α, remains unchanged with the number of training steps, and the value of the variable term, α, comprises a maximum value until a second predetermined number of training steps defining the stable phase, and

during the decreasing phase, the value of the variable term, α, decreases with the number of training steps according to a combination of the maximum value and a combined ratio of a maximum number of training steps, the current number of training steps, and the second predetermined number of training steps.

14 . A system for training a language model, the system comprising:

a computer processor;

a data repository in communication with the computer processor and storing:

a prompt,

a preferred output,

a non-preferred output,

a reference score,

a policy score,

a variable term, α, wherein α varies with a number of training steps performed,

a loss function, wherein:

the loss function comprises a combination of the policy score and the reference score,

the loss function further comprises a hyperparameter that modifies the combination of the policy score and the reference score, and

the variable term, α, varies with the number of training steps performed,

a loss output by executing the loss function;

a reference language model comprising a reference initial parameter and executable by the computer processor;

a training language model comprising an updatable parameter and executable by the computer processor;

an updated language model comprising the training language model having a final updated parameter and executable by the computer processor;

a training controller which, when executed by the computer processor, executes, a plurality of times, a training step of the number of training steps, wherein the training step comprises:

executing, on the prompt, the preferred output, and the non-preferred output, the reference language model to generate the reference score;

executing, on the prompt, the preferred output, and the non-preferred output, the training language model to generate the policy score;

executing the loss function to output the loss;

generating, from the loss, an update to the updatable parameter,

updating the training language model by adjusting the updatable parameter of the training language model according to the update, and

incrementing the number of trainings steps; and

a server controller executable by the computer processor to:

determine, after executing the training step the plurality of times, that convergence has occurred, and

return, after convergence, the updated language model as an improved language model.

15 . The system of claim 14 , wherein:

the training step improves an alignment of the improved language model to the preferred output over the non-preferred output, relative to the training language model, and

the training step increases a stability of the improved language model relative to the training language model.

16 . The system of claim 14 , wherein:

the training language model comprises a neural network having a plurality of training parameters connecting layers of nodes in the neural network and the reference language model comprises the neural network having a plurality of reference parameters connecting layers of nodes in the neural network,

the plurality of training parameters are permitted to change during the training step, and include the updatable parameter,

the plurality of reference parameters are unchangeable during the training step, and include the reference initial parameter,

the training language model, during the training step, outputs a training chosen score and a training reject score, and

the reference language model, during the training step, outputs a reference chosen score and a reference reject score.

17 . The system of claim 16 , the training step further comprises:

determining, from the training chosen score and the training reject score, the policy score; and

determining, from the reference chosen score and a reference reject score, the reference score.

18 . The system of claim 17 , wherein the loss function is based on the policy score, the reference score, and the hyperparameter.

19 . The system of claim 14 , wherein the variable term, α, comprises an increasing phase, a stable phase, and a decreasing phase, wherein:

during the increasing phase during the training step, a value of the variable term, α, increases with the number of training steps,

during the stable phase during the training step, the value of the variable term, α, remains unchanged with the number of training steps, and

during the decreasing phase during the training step, the value of the variable term, α, decreases with the number of training steps.

20 . A method for training a language model, the method comprising:

executing, a plurality of times, a training step, wherein the training step comprises:

executing, on a prompt, a preferred output, and a non-preferred output, a reference language model to generate a reference score;

executing, on the prompt, the preferred output, and the non-preferred output, a training language model to generate a policy score;

executing a loss function, wherein:

the loss function comprises a combination of the policy score and the reference score,

the loss function further comprises a hyperparameter that modifies the combination of the policy score and the reference score,

the hyperparameter comprises a variable term, α, wherein α varies with a number of training steps performed, including the training step, wherein the variable term, α, comprises an increasing phase, a stable phase, and a decreasing phase, wherein:

during the increasing phase, a value of the variable term, α, increases with the number of training steps,

during the stable phase, the value of the variable term, α, remains unchanged with the number of training steps, and

during the decreasing phase, the value of the variable term, α, decreases with the number of training steps, and

executing the loss function outputs a loss,

generating, from the loss, an updated parameter;

updating the training language model by adjusting an initial parameter of the training language model to the updated parameter, wherein updating generates an updated language model; and

incrementing the number of trainings steps;

determining, after executing the training step the plurality of times, that convergence has occurred; and

returning, after convergence, the updated language model as an improved language model, wherein:

the training step improves an alignment of the improved language model to the preferred output over the non-preferred output, relative to the training language model, and

the training step increases a stability of the improved language model relative to the training language model.