Instruction following in large language models to reduce computational resource consumption
Implementations relate to improving instruction following capabilities of large language models (LLMs) using instruction decomposition, self-evaluation, and optionally progressive refinement. Processor(s) of a system can: obtain natural language (NL) based input, generate a plurality of candidate responses and evaluate the candidate responses based on instructions included in the NL based input, using an LLM, and progressively refine the candidate responses until it is determined that one or more termination criteria are satisfied. In some implementations, the NL based input can be received from a client device. In these implementations, a given candidate response that is progressively refined can be rendered for presentation at the client device and responsive to the NL base input. In additional or alternative implementations, the NL based input can be obtained from database(s). In these implementations, a given candidate response that is progressively refined can be utilized in fine-tuning of the LLM.
1 . A method implemented by one or more processors, the method comprising:
fine-tuning, based on a plurality of training instances, a large language model (LLM); and
causing the LLM to be deployed, wherein causing the LLM to be deployed comprises:
receiving natural language (NL) based input associated with a client device, the NL based input including a set of instructions provided by a user of the client device;
generating a plurality of candidate LLM responses based on processing the NL based input using the LLM, wherein the plurality of candidate LLM responses together form a set of candidate LLM responses;
generating, for each of the plurality of candidate LLM responses, a corresponding critique response, the corresponding critique responses being generated based on processing, using the LLM, the plurality of candidate LLM responses and the set of instructions included in the NL based input;
determining, based on each of the corresponding critique responses, an indication of an extent to which the corresponding candidate LLM response complies with the set of instructions included in the NL based input, wherein the indication of the extent to which the corresponding candidate LLM response complies with the set of instructions is indicative of the instructions of the set of instructions the corresponding candidate LLM response is determined to comply with; and
in response to determining that a given candidate LLM response, from among the set of candidate LLM responses, complies with at least a threshold number of instructions of the set of instructions based on the corresponding critique responses:
selecting the given candidate LLM response from among the set of candidate LLM responses; and
causing the given candidate LLM response to be rendered at the client device.
2 . The method of claim 1 , further comprising:
progressively refining the candidate LLM responses until the given candidate LLM response complies with each instruction of the set of instructions included in the NL based input.
3 . The method of claim 1 , wherein the given candidate LLM response is selected from among the set of candidate LLM responses based on determining that the given candidate LLM response best complies with the set of instructions from among the set of candidate LLM responses.
4 . The method of claim 1 , wherein the given candidate LLM response is selected from among the set of candidate LLM responses based on searching a priority queue, wherein the priority queue comprises the set of candidate LLM responses, and wherein each of the set of candidate LLM responses in the priority queue is associated with a priority based on the corresponding critique response.
5 . The method of claim 1 , further comprising:
progressively refining the candidate LLM responses until one or more termination criteria are satisfied,
wherein the one or more termination criteria comprises one or more of: a threshold number of repetitions, a threshold execution time, a threshold number of processor cycles, or a threshold memory utilization, and
wherein, when the one or more termination criteria are satisfied, the given candidate LLM response is selected based on determining that the given candidate LLM response best complies with the set of instructions from among the set of candidate LLM responses.
6 . The method of claim 1 , further comprising:
progressively refining the candidate LLM responses until one or more termination criteria are satisfied,
wherein the one or more termination criteria comprises the threshold number of instructions of the set of instructions, and
wherein, when the one or more termination criteria are satisfied, the given candidate LLM response is selected based on determining that the given candidate LLM response best complies with the set of instructions from among the set of candidate LLM responses.
7 . The method of claim 1 , further comprising:
generating a plurality of refined candidate LLM responses, wherein generating the plurality of refined candidate LLM responses comprises:
generating a request for the LLM to refine a first candidate LLM response based on the corresponding critique response, wherein the request for the LLM to refine the first candidate LLM response comprises the indication of the instructions of the set of instructions that the first candidate LLM response was determined to violate based on the corresponding critique response; and
processing the request using the LLM to generate the plurality of refined LLM responses.
8 . The method of claim 1 , wherein generating the corresponding critique response for each of the plurality of candidate LLM responses comprises:
generating a request for the LLM to identify the set of instructions included in the NL based input and to determine the extent to which each of the candidate LLM responses complies with the set of instructions; and
processing the request and the NL based input using the LLM to identify the set of instructions included in the NL based input and to generate the corresponding critique responses.
9 . The method of claim 1 , further comprising:
determining whether to proceed with progressive refinement based on one or both of:
contextual data associated with the client device and/or the user of the client device, and
the NL based input.
10 . A method implemented by one or more processors, the method comprising:
fine-tuning, based on a plurality of training instances, a large language model (LLM); and
causing the LLM to be deployed, wherein causing the LLM to be deployed comprises:
obtaining natural language (NL) based input, the NL based input including a set of instructions provided by a user of a client device;
generating a plurality of candidate LLM responses based on processing the NL based input using the LLM, wherein the plurality of candidate LLM responses together form a set of candidate LLM responses;
generating, for each of the plurality of candidate LLM responses, a corresponding critique response, the corresponding critique responses being generated based on processing, using the LLM, the plurality of candidate LLM responses and the set of instructions included in the NL based input;
determining, based on each of the corresponding critique responses, an indication of an extent to which the corresponding candidate LLM response complies with the set of instructions included in the NL based input, wherein the indication of the extent to which the corresponding candidate LLM response complies with the set of instructions is indicative of the instructions of the set of instructions the corresponding candidate LLM response is determined to comply with; and
in response to determining that a given candidate LLM response, from among the set of candidate LLM responses, complies with at least a threshold number of instructions of the set of instructions based on the corresponding critique responses:
selecting the given candidate LLM response from among the set of candidate LLM responses; and
storing, as an additional training instance for further fine-tuning the LLM, the NL based input along with the given candidate LLM response.
11 . The method of claim 10 , further comprising:
progressively refining the candidate LLM responses until the given candidate LLM response complies with each instruction of the set of instructions included in the NL based input.
12 . The method of claim 10 , wherein the given candidate LLM response is selected from among the set of candidate LLM responses based on determining that the given candidate LLM response best complies with the set of instructions from among the set of candidate LLM responses.
13 . The method of claim 10 , wherein the given candidate LLM response is selected from among the set of candidate LLM responses based on searching a priority queue, wherein the priority queue comprises the set of candidate LLM responses, and wherein each of the set of candidate LLM responses in the priority queue is associated with a priority based on the corresponding critique response.
14 . The method of claim 10 , further comprising:
progressively refining the candidate LLM responses until one or more termination criteria are satisfied,
wherein the one or more termination criteria comprises one or more of: a threshold number of repetitions, a threshold execution time, a threshold number of processor cycles, or a threshold memory utilization, and
wherein, when the one or more termination criteria are satisfied, the given candidate LLM response is selected based on determining that the given candidate LLM response best complies with the set of instructions from among the set of candidate LLM responses.
15 . The method of claim 10 , further comprising:
progressively refining the candidate LLM responses until one or more termination criteria are satisfied,
wherein the one or more termination criteria comprises the threshold number of instructions of the set of instructions, and
wherein, when the one or more termination criteria are satisfied, the given candidate LLM response is selected based on determining that the given candidate LLM response best complies with the set of instructions from among the set of candidate LLM responses.
16 . The method of claim 10 , further comprising:
generating a plurality of refined candidate LLM responses, wherein generating the plurality of refined candidate LLM responses comprises:
generating a request for the LLM to refine a first candidate LLM response based on the corresponding critique response, wherein the request for the LLM to refine the first candidate LLM response comprises the indication of the instructions of the set of instructions that the first candidate LLM response was determined to violate based on the corresponding critique response; and
processing the request using the LLM to generate the plurality of refined LLM responses.
17 . The method of claim 10 , wherein generating the corresponding critique response for each of the plurality of candidate LLM responses comprises:
generating a request for the LLM to identify the set of instructions included in the NL based input and to determine the extent to which each of the candidate LLM responses complies with the set of instructions; and
processing the request and the NL based input using the LLM to identify the set of instructions included in the NL based input and to generate the corresponding critique responses.
18 . A system comprising:
at least one processor; and
memory storing instructions that, when executed, cause the at least one processor to be operable to:
fine-tune, based on a plurality of training instances, a large language model (LLM); and
cause the LLM to be deployed, wherein the instructions to cause the LLM to be deployed comprise instructions to:
receive natural language (NL) based input associated with a client device, the NL based input including a set of instructions provided by a user of the client device;
generate a plurality of candidate LLM responses based on processing the NL based input using the LLM, wherein the plurality of candidate LLM responses together form a set of candidate LLM responses;
generate, for each of the plurality of candidate LLM responses, a corresponding critique response, the corresponding critique responses being generated based on processing, using the LLM, the plurality of candidate LLM responses and the set of instructions included in the NL based input;
determine, based on each of the corresponding critique responses, an indication of an extent to which the corresponding candidate LLM response complies with the set of instructions included in the NL based input, wherein the indication of the extent to which the corresponding candidate LLM response complies with the set of instructions is indicative of the instructions of the set of instructions the corresponding candidate LLM response is determined to comply with; and
in response to determining that a given candidate LLM response, from among the set of candidate LLM responses, complies with at least a threshold number of instructions of the set of instructions based on the corresponding critique responses:
select the given candidate LLM response from among the set of candidate LLM responses; and
cause the given candidate LLM response to be rendered at the client device.
19 . The system of claim 18 , wherein the instructions further cause the at least one processor to be operable to:
progressively refine the candidate LLM responses until the given candidate LLM response complies with each instruction of the set of instructions included in the NL based input.
20 . The system of claim 18 , wherein the given candidate LLM response is selected from among the set of candidate LLM responses based on determining that the given candidate LLM response best complies with the set of instructions from among the set of candidate LLM responses.