Application specific auto-evaluation for large language models (LLMs)
In one embodiment, a non-transitory computer-readable media stores instructions executable by processors for generating a prompt configured for eliciting outputs from large language models (LLMs) based on information associated with a task, inputting the prompt to a first LLM configured to output a response based on processing the prompt, determining metrics for evaluating the first LLM based on the task, wherein each of the metrics is associated with a scoring guideline, generating metric prompts based on the respective metrics and the scoring guidelines associated with the respective metrics, inputting the response and the metric prompts to second LLMs configured to output scores corresponding to the respective metrics based on processing the response and the metric prompts, and generating an analysis report based on the metrics and their corresponding scores.
1 . One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause performance of:
generating, based on information associated with a first task, a prompt configured for eliciting outputs from large language models (LLMs);
inputting the prompt to a first LLM, wherein the first LLM is configured to output a response based on processing the prompt;
determining, based on the first task, one or more metrics for evaluating the first LLM, wherein each of the metrics is associated with a scoring guideline;
generating one or more metric prompts based on the one or more respective metrics and the one or more scoring guidelines associated with the one or more respective metrics;
inputting the response and the one or more metric prompts to one or more second LLMs, wherein the one or more second LLMs are configured to output one or more scores corresponding to the one or more respective metrics based on processing the response and the metric prompts; and
generating an analysis report based on the one or more metrics and their corresponding scores.
2 . The media of claim 1 , wherein the instructions when executed by the processors, cause further performance of:
generating one or more examples for each of the one or more metrics; and
injecting the examples to each of the metric prompts associated with the respective metric.
3 . The media of claim 2 , wherein each of the examples comprises one or more of:
a demo;
a reference score for the demo; or
an explanation for the reference score.
4 . The media of claim 1 , wherein the instructions when executed by the processors, cause further performance of:
determining one or more thresholds for the one or more metrics, respectively;
comparing the one or more scores with the one or more thresholds, respectively; and
determining whether the first LLM passes or fails the evaluation based on the comparison.
5 . The media of claim 4 , wherein the instructions when executed by the processors, cause further performance of:
determining all the metrics exceed their respective thresholds;
based at least in part on said determining all the metrics exceed their respective thresholds, determining the first LLM passes the evaluation; and
wherein the analysis report comprises an indication that the first LLM passes the evaluation.
6 . The media of claim 4 , wherein the instructions when executed by the processors, cause further performance of:
determining one or more of the metrics are lower than their corresponding thresholds;
based at least in part on said determining one or more of the metrics are lower than their corresponding thresholds, determining the first LLM fails the evaluation; and
wherein the analysis report comprises an indication that the first LLM fails the evaluation.
7 . The media of claim 4 , wherein the one or more thresholds are determined based on one or more requirements associated with the first task.
8 . The media of claim 4 , wherein the instructions when executed by the processors, cause further performance of:
inputting the prompt to the first LLM for a plurality of respective times, wherein the first LLM is configured to output a plurality of respective responses based on processing the prompt for the plurality of respective times;
determining, by the one or more second LLMs, a plurality of sets of one or more scores corresponding to the one or more respective metrics based on processing the plurality of response and the metric prompts; and
calculating a pass-rate for the first LLM based on determining whether the first LLM passes or fails the evaluation for each of the plurality of times;
wherein the analysis report comprises the pass-rate.
9 . The media of claim 1 , wherein the instructions when executed by the processors, cause further performance of:
generating, based on one or more of the metric prompts, one or more metric functions configured to be re-used for one or more second tasks.
10 . The media of claim 1 , wherein generating the prompt is based on a prompt template and one or more input variables associated with the first task.
11 . The media of claim 10 , wherein each of the input variables comprises a pair of variable name and value.
12 . The media of claim 1 , wherein the metrics comprise one or more of tone, natural language quality, variety, repetitiveness, content classification, content relevance, JSON format, or length.
13 . The media of claim 1 , wherein one or more of the metrics are shareable between the first task and one or more second tasks.
14 . The media of claim 1 , wherein the one or more second LLMs are based on a same model.
15 . The media of claim 1 , wherein one or more of the second LLMs are based on different models.
16 . A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions, when executed using the processors, cause the system to execute:
generating, based on information associated with a first task, a prompt configured for eliciting outputs from large language models (LLMs);
inputting the prompt to a first LLM, wherein the first LLM is configured to output a response based on processing the prompt;
determining, based on the first task, one or more metrics for evaluating the first LLM, wherein each of the metrics is associated with a scoring guideline;
generating one or more metric prompts based on the one or more respective metrics and the one or more scoring guidelines associated with the one or more respective metrics;
inputting the response and the one or more metric prompts to one or more second LLMs, wherein the one or more second LLMs are configured to output one or more scores corresponding to the one or more respective metrics based on processing the response and the metric prompts; and
generating an analysis report based on the one or more metrics and their corresponding scores.
17 . The system of claim 16 , wherein the instructions when executed using the processors, cause the processors to further execute:
generating one or more examples for each of the one or more metrics; and
injecting the examples to each of the metric prompts associated with the respective metric.
18 . The system of claim 16 , wherein the instructions when executed using the processors, cause the processors to further execute:
determining one or more thresholds for the one or more metrics, respectively;
comparing the one or more scores with the one or more thresholds, respectively; and
determining whether the first LLM passes or fails the evaluation based on the comparison.
19 . A method comprising, by one or more computing systems:
generating, based on information associated with a first task, a prompt configured for eliciting outputs from large language models (LLMs);
inputting the prompt to a first LLM, wherein the first LLM is configured to output a response based on processing the prompt;
determining, based on the first task, one or more metrics for evaluating the first LLM, wherein each of the metrics is associated with a scoring guideline;
generating one or more metric prompts based on the one or more respective metrics and the one or more scoring guidelines associated with the one or more respective metrics;
inputting the response and the one or more metric prompts to one or more second LLMs, wherein the one or more second LLMs are configured to output one or more scores corresponding to the one or more respective metrics based on processing the response and the metric prompts; and
generating an analysis report based on the one or more metrics and their corresponding scores.
20 . The method of claim 19 , further comprising:
generating one or more examples for each of the one or more metrics; and
injecting the examples to each of the metric prompts associated with the respective metric.