Computationally efficient reasoning language models

1 . A method of increasing a computational efficiency of a pretrained reasoning language model (RLM) comprising a plurality of nodes in a plurality of layers, and further comprising a plurality of weights indicating relationships among nodes between any two layers of the plurality of layers, the method comprising:

executing, in a plurality of executions, the RLM on a query to generate a plurality of answers having a plurality of lengths;

executing a classification model on the plurality of answers to identify a brief answer comprising a first number of tokens and at least one detailed answer comprising a second number of tokens greater than the first number;

labeling the brief answer as a first label and labeling the at least one detailed answer as at least one second label;

generating a loss function for a reinforcement learning algorithm, wherein the loss function is generated using a combination of the first label, the at least one second label, the brief answer, and the at least one detailed answer;

training the RLM using the reinforcement learning algorithm and the loss function, wherein training modifies the plurality of weights and the relationships among the nodes, and wherein:

the loss function, using the combination of the first label, the at least one second label, the brief answer, and the at least one detailed answer, rewards the brief answer and penalizes the at least one detailed answer;

training generates a tuned RLM trained to generate fewer tokens in response to an input, relative to the pretrained RLM, and

the tuned RLM comprises a higher computational efficiency than the pretrained RLM;

receiving a new input;

executing the tuned RLM on the new input, wherein, during executing, the tuned RLM generates a first intermediate output;

executing, prior to generating a next intermediate output, the classification model on the first intermediate output to generate a predicted complexity of a final output of the tuned RLM;

comparing the predicted complexity to a complexity threshold;

determining that the predicted complexity fails to satisfy the complexity threshold;

commanding, responsive to determining that the predicted complexity fails to satisfy the complexity threshold, the tuned RLM to cease generating intermediate outputs; and

returning the first intermediate output as the final output.

2 . The method of claim 1 , wherein the loss function comprises a reward model that rewards the brief answer and penalizes the at least one detailed answer.

3 . The method of claim 1 , further comprising:

identifying, prior to executing the classification model, incorrect answers from the plurality of answers; and

eliminating, prior to executing the classification model, the incorrect answers from the plurality of answers.

4 . The method of claim 1 , wherein the reinforcement learning algorithm is selected from the group consisting of a direct preference optimization algorithm, a monolithic preference optimization without reference model algorithm, and an odds ratio preference optimization algorithm, a Kahneman-Tversky optimization algorithm.

5 . A method comprising:

receiving a query to a reasoning language model (RLM), wherein the RLM is trained using a loss function that combines:

a first label representing a brief answer,

at least one second label representing at least one detailed answer, the brief answer, and

the at least one detailed answer,

wherein during training the loss function further rewarded the brief answer and penalized the at least one detailed answer;

generating, by the RLM in response to the query, a first token of a final output of the RLM;

generating, by the RLM after the first token, an intermediate token of the final output, wherein the intermediate token comprises a token generated prior to a final token output by the RLM;

executing, prior to generating a next intermediate token of the final output, a classification model on the intermediate token, to generate a predicted complexity of the final output of the RLM, wherein the predicted complexity comprises a prediction whether the next intermediate token of the final output adds to at least one of an accuracy or a diversity of reasoning of a combination of the first token and the intermediate token;

determining that the predicted complexity fails to satisfy a complexity threshold;

commanding, responsive to determining that the predicted complexity fails to satisfy the complexity threshold, the RLM to terminate execution immediately after generating the intermediate token and prior to generating the final output; and

returning, responsive to terminating execution of the RLM, the first token and the intermediate token as a revised final output.

6 . The method of claim 5 , further comprising:

training the classification model by:

receiving a pair of answers, wherein the pair of answers comprises a brief correct answer and a detailed correct answer comprising a greater number of tokens than the brief correct answer,

executing the classification model on the pair of answers to generate an intermediate prediction that the detailed correct answer is a more complete answer than the brief correct answer,

comparing the intermediate prediction to a correct prediction that the detailed correct answer is more complete than the brief correct answer,

generating, using the pair of answers, the intermediate prediction, and the correct prediction, the loss function, and

updating weights of the classification model using the loss function.

7 . The method of claim 5 , wherein training the classification model repeats until convergence.