Confidence-based reward for group relative policy optimization in language models
Certain aspects of the disclosure provide a method for training a language model (LM) including: generating, using an LM, one or more outputs; computing a confidence score of an output of the one or more outputs based on a perplexity value of the output; determining, by a group relative policy optimization (GRPO)-based model, that the output is: associated with a correct status based on a reference policy; and associated with an uncertain status based on the confidence score and a threshold; determining, by the GRPO-based model, an increased reward value for the output that is associated with the correct status and the uncertain status based at least in part on a base reward value and the confidence score; causing, by the GRPO-based model, a reinforcement of the output using the increased reward value; and training the LM in accordance with the reinforcement of the output.
1 . A method, comprising:
generating, using a language model (LM), one or more outputs;
computing a confidence score of an output of the one or more outputs based on a perplexity value of the output;
determining, by a group relative policy optimization (GRPO)-based model, that the output is:
associated with a correct status based on a reference policy; and
associated with an uncertain status based on the confidence score and a threshold;
determining, by the GRPO-based model, an increased reward value for the output that is associated with the correct status and the uncertain status based at least in part on a base reward value and the confidence score;
causing, by the GRPO-based model, a reinforcement of the output using the increased reward value; and
training, by the GRPO-based model, the LM in accordance with the reinforcement of the output, the training of the LM comprising causing one or more actions to be performed based on the reinforcement, the one or more actions comprising transmitting data to update one or more policies associated with the LM.
2 . The method of claim 1 , wherein the perplexity value is a measure of uncertainty of the output, and the confidence score is a measure of confidence of the output.
3 . The method of claim 1 , wherein the confidence score is represented as an inverse of the perplexity value.
4 . The method of claim 1 , wherein determining that the output is associated with the correct status comprises:
determining that the output is associated with the correct status with respect to an input to the LM based on a validation process, the validation process comprising at least one of a factual accuracy check of the output, a labeling process, or comparing a value of the output and a value of the reference policy.
5 . The method of claim 1 , wherein determining that the output is associated with the uncertain status comprises:
determining that confidence score of the output is below the threshold.
6 . The method of claim 1 , wherein the increased reward value is based on the base reward value of the output, a calibration factor, and the confidence score.
7 . The method of claim 1 , comprising:
reinforcing, based on another output being at least one of associated with an incorrect status or not associated with the uncertain status, the other output without increasing the base reward value.
8 . The method of claim 1 , wherein the LM is trained when the LM is in at least one of an offline mode or an online mode.
9 . A method, comprising:
generating, using a language model (LM), a plurality of outputs;
computing a confidence score of each output of the plurality of outputs based on a respective perplexity value of the plurality of outputs;
determining, by a group relative policy optimization (GRPO)-based model, that a set of outputs of the plurality of outputs are:
associated with a correct status based on a reference policy; and
associated with an uncertain status based on the confidence score satisfying a threshold for the set of outputs;
determining, by the GRPO-based model, an increased reward value for the set of outputs that are associated with the correct status and the uncertain status based at least in part on a base reward value and the confidence score;
causing, by the GRPO-based model, a reinforcement of the set of outputs using the increased reward value; and
performing, by the GRPO-based model, one or more actions based on the reinforcement, the one or more actions comprising transmitting data to update one or more policies associated with the LM.
10 . The method of claim 9 , wherein the perplexity value is a measure of uncertainty of the plurality of outputs, and the confidence score is a measure of confidence of the plurality of outputs.
11 . The method of claim 9 , wherein the confidence score is represented as an inverse of the perplexity value.
12 . The method of claim 9 , wherein determining that the set of outputs is associated with the correct status comprises:
determining that the set of outputs is associated with the correct status with respect to an input to the LM based on a validation process, the validation process comprising at least one of a factual accuracy check of the set of outputs, a labeling process, or comparing a value of the set of outputs and a value of the reference policy.
13 . The method of claim 9 , wherein determining that the set of outputs is associated with the uncertain status comprises:
determining that confidence score of the set of outputs is below the threshold.
14 . The method of claim 9 , wherein the increased reward value is based on the base reward value of the set of outputs, a calibration factor, and the confidence score.
15 . The method of claim 9 , comprising:
reinforcing, based on another set of outputs being at least one of associated with an incorrect status or not associated with the uncertain status, the other set of outputs without increasing the base reward value.
16 . The method of claim 9 , further comprising:
training the LM in accordance with the reinforcement of the set of outputs.
17 . The method of claim 9 , further comprising:
updating one or more policies associated with the LM, the updated one or more policies triggering the LM to select an output of the set of outputs as an answer to an input to the LM.
18 . The method of claim 17 , wherein the selected output is discardable as the answer by the LM if the output is not reinforced using the increased reward value.
19 . An apparatus comprising a processing system that includes one or more memories and one or more processors coupled to the one or more memories, the processing system configured to cause the apparatus to:
generate, using a large language model (LLM), one or more outputs;
compute a confidence score of an output of the one or more outputs based on a perplexity value of the output;
determine, by a group relative policy optimization (GRPO)-based model, that the output is:
associated with a correct status based on a reference policy; and
associated with an uncertain status based on the confidence score and a threshold;
determine, by the GRPO-based model, an increased reward value for the output that is associated with the correct status and the uncertain status based at least in part on a base reward value and the confidence score;
cause, by the GRPO-based model, a reinforcement of the output using the increased reward value; and
train, by the GRPO-based model, the LLM in accordance with the reinforcement of the output, wherein to train the LLM the processing system is further configured to cause the apparatus to cause one or more actions to be performed based on the reinforcement, the one or more actions comprising transmitting data to update one or more policies associated with the LM.
20 . The apparatus of claim 19 , wherein the processing system is further configured to cause the apparatus to:
reinforce, based on another output being at least one of associated with an incorrect status or not associated with the uncertain status, the other output without increasing the base reward value.