Generating performance metrics to facilitate large language model operations

1 . A system comprising:

a processing device; and

a memory device that stores instructions executable by the processing device for causing the processing device to perform operations comprising:

receiving a first structured language file generated by a large language model, wherein the first structured language file comprises a set of generated keys associated with a resource and a set of generated values associated with the resource;

receiving a second structured language file generated by a user, wherein the second structured language file comprises a set of test keys associated with the resource and a set of test values associated with the resource;

extracting, from a schema for the resource, a set of resource keys;

determining, based on a comparison of the set of generated keys and the set of resource keys, a valid key total representative of a number of valid keys in the set of generated keys;

determining, based on a comparison of the set of generated values and the set of test values, a valid value total representative of a number of valid values in the set of generated values;

generating a first plurality of performance metrics for the first structured language file generated by the large language model and transmitting the first plurality of performance metrics to a user device, wherein each performance metric in the first plurality of performance metrics depends on the valid key total or the valid value total;

determining, based on the first plurality of performance metrics, a model training recommendation;

executing a training process of the large language model that uses training data selected based on the model training recommendation to modify one or more internal data structures of the large language model; and

subsequent to executing the training process of the large language model, deploying and executing the large language model on the user device and generating, by the large language model executing on the user device, a third structured language file, wherein a second plurality of performance metrics indicate that the third structured language file is more accurate than the first structured language file.

2 . The system of claim 1 , wherein the first plurality of performance metrics comprise a hallucination score, and wherein the operation of generating the first plurality of performance metrics comprises computing the hallucination score by:

determining a resource key total representative of a number of keys in the set of resource keys;

determining a generated key total representative of a number of keys in the set of generated keys;

computing a difference between the generated key total and the valid key total to produce an invalid key total; and

computing a ratio of the invalid key total to the resource key total.

3 . The system of claim 1 , wherein the first plurality of performance metrics comprise a key error score and a value error score, and wherein the operation of generating the first plurality of performance metrics comprises:

computing the key error score based on a number of keys in the set of test keys and the valid key total; and

computing the value error score based on a number of values in the set of test values and the valid value total.

4 . The system of claim 1 , wherein the set of resource keys comprise a subset of fundamental resource keys, wherein the first plurality of performance metrics comprise a verbosity score, and wherein the operation of generating the first plurality of performance metrics comprises computing the verbosity score by:

determining a resource key total representative of a number of keys in the set of resource keys;

determining a fundamental key total representative of a number of keys in the subset of fundamental resource keys;

computing a first difference between the valid key total and the fundamental key total;

computing a second difference between the valid key total and a number of keys in the set of test keys;

computing a third difference between the first difference and the second difference; and

computing a ratio of the third difference to the resource key total.

5 . The system of claim 1 , wherein the set of resource keys comprise a subset of fundamental resource keys, wherein the first plurality of performance metrics comprise a minimum valid generation score, and wherein the operation of generating the first plurality of performance metrics comprises computing the minimum valid generation score by:

determining a subset of generated keys in the set of generated keys that match a key in the subset of fundamental resource keys;

determining a fundamental key total representative of a number of keys in the subset of fundamental resource keys;

computing a difference between a number of keys in the subset of generated keys and the fundamental key total; and

computing a ratio of the difference to the fundamental key total.

6 . The system of claim 1 , wherein the operations further comprise:

mapping the first structured language file to a classification group of a plurality of classification groups, wherein the plurality of classification groups comprise a true positive group, a true negative group, a false positive group, and a false negative group.

7 . The system of claim 1 , wherein the first plurality of performance metrics comprise a test complexity score, and wherein the operation of generating the first plurality of performance metrics comprises computing the test complexity score by:

determining a resource key total representative of a number of keys in the set of resource keys; and

computing a ratio of a number of keys in the set of test keys to the resource key total.

8 . A computer-implemented method comprising:

receiving a first structured language file generated by a large language model, wherein the first structured language file comprises a set of generated keys associated with a resource and a set of generated values associated with the resource;

receiving a second structured language file generated by a user, wherein the second structured language file comprises a set of test keys associated with the resource and a set of test values associated with the resource;

extracting, from a schema for the resource, a set of resource keys;

determining, based on a comparison of the set of generated keys and the set of resource keys, a valid key total representative of a number of valid keys in the set of generated keys;

determining, based on a comparison of the set of generated values and the set of test values, a valid value total representative of a number of valid values in the set of generated values;

generating a first plurality of performance metrics for the first structured language file generated by the large language model and transmitting the first plurality of performance metrics to a user device, wherein each performance metric in the first plurality of performance metrics depends on the valid key total or the valid value total;

determining, based on the first plurality of performance metrics, a model training recommendation;

executing a training process of the large language model that uses training data selected based on the model training recommendation to modify one or more internal data structures of the large language model; and

subsequent to executing the training process of the large language model, deploying and executing the large language model on the user device and generating, by the large language model executing on the user device, a third structured language file, wherein a second plurality of performance metrics indicate that the third structured language file is more accurate than the first structured language file.

9 . The computer-implemented method of claim 8 , wherein the first plurality of performance metrics comprise a hallucination score, and wherein generating the first plurality of performance metrics comprises computing the hallucination score by:

determining a resource key total representative of a number of keys in the set of resource keys;

determining a generated key total representative of a number of keys in the set of generated keys;

computing a difference between the generated key total and the valid key total to produce an invalid key total; and

computing a ratio of the invalid key total to the resource key total.

10 . The computer-implemented method of claim 8 , wherein the first plurality of performance metrics comprise a key error score and a value error score, and wherein generating the first plurality of performance metrics comprises:

computing the key error score based on a number of keys in the set of test keys and the valid key total; and

computing the value error score based on a number of values in the set of test values and the valid value total.

11 . The computer-implemented method of claim 8 , wherein the set of resource keys comprise a subset of fundamental resource keys, wherein the first plurality of performance metrics comprise a verbosity score, and wherein generating the first plurality of performance metrics comprises computing the verbosity score by:

determining a resource key total representative of a number of keys in the set of resource keys;

determining a fundamental key total representative of a number of keys in the subset of fundamental resource keys;

computing a first difference between the valid key total and the fundamental key total;

computing a second difference between the valid key total and a number of keys in the set of test keys;

computing a third difference between the first difference and the second difference; and

computing a ratio of the third difference to the resource key total.

12 . The computer-implemented method of claim 8 , wherein the set of resource keys comprise a subset of fundamental resource keys, wherein the first plurality of performance metrics comprise a minimum valid generation score, and wherein generating the first plurality of performance metrics comprises computing the minimum valid generation score by:

determining a subset of generated keys in the set of generated keys that match a key in the subset of fundamental resource keys;

determining a fundamental key total representative of a number of keys in the subset of fundamental resource keys;

computing a difference between a number of keys in the subset of generated keys and the fundamental key total; and

computing a ratio of the difference to the fundamental key total.

13 . The computer-implemented method of claim 8 , further comprising:

mapping the first structured language file to a classification group of a plurality of classification groups, wherein the plurality of classification groups comprise a true positive group, a true negative group, a false positive group, and a false negative group.

14 . A non-transitory computer-readable medium storing instructions that are executable by a processing device for causing the processing device to perform operations comprising:

receiving a first structured language file generated by a large language model, wherein the first structured language file comprises a set of generated keys associated with a resource and a set of generated values associated with the resource;

receiving a second structured language file generated by a user, wherein the second structured language file comprises a set of test keys associated with the resource and a set of test values associated with the resource;

extracting, from a schema for the resource, a set of resource keys;

determining, based on a comparison of the set of generated keys and the set of resource keys, a valid key total representative of a number of valid keys in the set of generated keys;

determining, based on a comparison of the set of generated values and the set of test values, a valid value total representative of a number of valid values in the set of generated values;

generating a first plurality of performance metrics for the first structured language file generated by the large language model and transmitting the first plurality of performance metrics to a user device, wherein each performance metric in the first plurality of performance metrics depends on the valid key total or the valid value total;

determining, based on the first plurality of performance metrics, a model training recommendation;

executing a training process of the large language model that uses training data selected based on the model training recommendation to modify one or more internal data structures of the large language model; and

subsequent to executing the training process of the large language model, deploying and executing the large language model on the user device and generating, by the large language model executing on the user device, a third structured language file, wherein a second plurality of performance metrics indicate that the third structured language file is more accurate than the first structured language file.

15 . The non-transitory computer-readable medium of claim 14 , wherein the first plurality of performance metrics comprise a hallucination score, and wherein the operation of generating the first plurality of performance metrics comprises computing the hallucination score by:

determining a resource key total representative of a number of keys in the set of resource keys;

determining a generated key total representative of a number of keys in the set of generated keys;

computing a difference between the generated key total and the valid key total to produce an invalid key total; and

computing a ratio of the invalid key total to the resource key total.

16 . The non-transitory computer-readable medium of claim 14 , wherein the first plurality of performance metrics comprise a key error score and a value error score, and wherein the operation of generating the first plurality of performance metrics comprises:

computing the key error score based on a number of keys in the set of test keys and the valid key total; and

computing the value error score based on a number of values in the set of test values and the valid value total.

17 . The non-transitory computer-readable medium of claim 14 , wherein the set of resource keys comprise a subset of fundamental resource keys, wherein the first plurality of performance metrics comprise a verbosity score, and wherein the operation of generating the first plurality of performance metrics comprises computing the verbosity score by:

determining a resource key total representative of a number of keys in the set of resource keys;

determining a fundamental key total representative of a number of keys in the subset of fundamental resource keys;

computing a first difference between the valid key total and the fundamental key total;

computing a second difference between the valid key total and a number of keys in the set of test keys;

computing a third difference between the first difference and the second difference; and

computing a ratio of the third difference to the resource key total.