Evaluating the performance of a large language model

1 . A method, comprising:

receiving a query;

evaluating a performance of a large language model that generated a query response based on the received query, wherein evaluating the performance of the large language model includes determining a reference dependent evaluator score and/or a reference free evaluator score, wherein the reference free evaluator score is based on a coherence value, a relevance value, a consistency value, and a fluency value, wherein the reference free evaluator score is determined in part by providing a second large language model a prompt that includes a chain of thoughts that will instruct the second large language model on how to grade the query response, wherein the prompt indicates a corresponding range of scores for the coherence value, the relevance value, the consistency value, and the fluency value;

requesting one or more documents used to generate the received query to be updated based on the reference dependent evaluator score associated with the query response;

receiving the one or more updated documents; and

storing the one or more updated documents, wherein the one or more updated documents are utilized by the large language model to generate one or more subsequent query responses for one or more subsequent queries.

2 . The method of claim 1 , wherein the reference dependent evaluator score is determined by comparing a token-by-token embedding of a retrieved context included in the query response to a token embedding of the query.

3 . The method of claim 1 , wherein the reference dependent evaluator score is a Bidirectional Encoder Representations from Transformers (BERT) score.

4 . The method of claim 1 , wherein the prompt indicates a scoring function to weigh the coherence value, the relevance value, the consistency value, and the fluency value.

5 . The method of claim 4 , wherein the reference free evaluator score is a sum of the coherence value, the relevance value, the consistency value, and the fluency value, wherein the coherence value, the relevance value, the consistency value, and the fluency value have a corresponding weight.

6 . The method of claim 1 , wherein evaluating the performance of the large language model includes comparing the reference dependent evaluator score to a first threshold.

7 . The method of claim 6 , wherein the first threshold is a 50th percentile of reference dependent evaluator scores.

8 . The method of claim 6 , wherein evaluating the performance of the large language model includes assigning the query response a resultant score of 0 in response to the reference dependent evaluator score being less than the first threshold.

9 . The method of claim 8 , wherein the one or more documents are associated with a knowledge document gap in the query response.

10 . The method of claim 6 , wherein evaluating the performance of the large language model includes determining the reference free evaluator score in response to the reference dependent score being greater than or equal to the first threshold.

11 . The method of claim 10 , wherein evaluating the performance of the large language model includes comparing the reference dependent evaluator score to a second threshold.

12 . The method of claim 11 , wherein the second threshold is a 75th percentile of reference dependent evaluator scores.

13 . The method of claim 11 , wherein the reference dependent evaluator score is assigned a value of 1 in response to the reference dependent evaluator score being greater than the second threshold.

14 . The method of claim 13 , wherein evaluating the performance of the large language model includes determining a resultant score for the query response based on the reference dependent evaluator score and the reference free evaluator score.

15 . The method of claim 11 , wherein in response to the reference dependent evaluator score not being greater than the second threshold, evaluating the performance of the large language model further includes:

determining a resultant score for the query response based on the reference dependent evaluator score and the reference free evaluator score; and

requesting a reviewer to review the query response.

16 . A system, comprising:

a processor configured to:

receive a query;

evaluate a performance of a large language model that generated a query response based on the received query, wherein evaluating the performance of the large language model includes determining a reference dependent evaluator score and/or a reference free evaluator score, wherein the reference free evaluator score is based on a coherence value, a relevance value, a consistency value, and a fluency value, wherein the reference free evaluator score is determined in part by providing a second large language model a prompt that includes a chain of thoughts that will instruct the second large language model on how to grade the query response, wherein the prompt indicates a corresponding range of scores for the coherence value, the relevance value, the consistency value, and the fluency value;

request one or more documents used to generate the received query to be updated based on the reference dependent evaluator score associated with the query response;

receive the one or more updated documents; and

store the one or more updated documents, wherein the one or more updated documents are utilized by the large language model to generate one or more subsequent query responses for one or more subsequent queries; and

a memory coupled to the processor and configured to provide the processor with instructions.

17 . A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:

receiving a query;

evaluating a performance of a large language model that generated a query response based on the received query, wherein evaluating the performance of the large language model includes determining a reference dependent evaluator score and/or a reference free evaluator score, wherein the reference free evaluator score is based on a coherence value, a relevance value, a consistency value, and a fluency value, wherein the reference free evaluator score is determined in part by providing a second large language model a prompt that includes a chain of thoughts that will instruct the second large language model on how to grade the query response, wherein the prompt indicates a corresponding range of scores for the coherence value, the relevance value, the consistency value, and the fluency value;

requesting one or more documents used to generate the received query to be updated based on the reference dependent evaluator score associated with the query response;

receiving the one or more updated documents; and

storing the one or more updated documents, wherein the one or more updated documents are utilized by the large language model to generate one or more subsequent query responses for one or more subsequent queries.

18 . The system of claim 16 , wherein the reference dependent evaluator score is determined by comparing a token-by-token embedding of a retrieved context included in the query response to a token embedding of the query.

19 . The system of claim 16 , wherein to evaluate the performance of the large language model, the processor is configured to compare the reference dependent evaluator score to a first threshold.

20 . The system of claim 19 , wherein to evaluate the performance of the large language model, the processor is further configured to determine the reference free evaluator score in response to the reference dependent score being greater than or equal to the first threshold.