Modular framework for evaluating language models
Aspects of the invention provide a method, computer system, and computer program product for evaluating language model responses. A set of user-defined objectives for evaluating model responses generated by a large language model is received. The objectives include semantic accuracy, grammatical soundness, and expert alignment. A set of evaluation metrics are selected from a predefined library based on the user-defined objectives. Evaluation scores are determined for model responses generated by a large language model using the selected set of metrics. The evaluation scores include a semantic score and a lexical score. A composite score is calculated from the evaluation scores. The composite score is adjusted for divergences between the semantic score and the lexical score. A set of hyperparameters of the large language models is tuned based on the composite score. The tuning customizes the large language model according to the user-defined objectives.
1 . A method for improving a performance of a large language model with respect to a set of user-defined objectives, the method comprising:
receiving the set of user-defined objectives for evaluating model responses generated by the large language model, wherein the set of user-defined objectives include at least one from semantic accuracy, grammatical soundness, and expert alignment;
based on the user-defined objectives, selecting a set of evaluation metrics from a predefined library;
determining, using the selected set of evaluation metrics, evaluation scores for model responses generated by a large language model, wherein the evaluation scores include a semantic score and a lexical score;
calculating a composite score from the evaluation scores, wherein the composite score is adjusted for divergences between the semantic score and the lexical score, wherein the composite score is calculated as:
S
comp
=
(
α
*
(
S
lex
+
b
lex
)
p
lex
+
(
1
-
α
)
*
α
*
(
S
sem
+
b
sem
)
p
sem
)
1
P
1
+
λ
×
penalty
wherein:
S comp is the composite similarity:
α is a weighted hyperparameter;
S lex is a lexical similarity;
b lex is a lexical bias;
p lex is a lexical tuning hyperparameter;
S sem is the semantic similarity;
b sem is a semantic bias;
p sem is a semantic tuning hyperparameter;
p is a mean behavior scaling factor;
λ is a penalty scaling factor; and
penalty is corrective factor for S lex -S sem divergence; and
tuning a set of hyperparameters of the large language models based on the composite score, wherein the tuning customizes the large language model according to the user-defined objectives by aligning the performance of the large language model according to the user-defined objectives.
2 . The method of claim 1 , further comprising:
collecting a set of evaluation data, wherein the evaluation data includes responses from subject matter experts and the model responses generated by the large language models;
preprocessing the collected evaluation data, wherein the preprocessing includes one or more natural language processing techniques to normalize and tokenize the evaluation data.
3 . The method of claim 1 , wherein the semantic score is calculated from evaluation metrics selected from a ROGUE score, a BLEU score, and combinations thereof.
4 . The method of claim 1 , wherein the lexical score is calculated from evaluation metrics selected from a BERT-Score, a BART-Score, a METEOR score, and combinations thereof.
5 . The method of claim 1 , wherein tuning the set of hyperparameters further comprises:
applying one or more optimization algorithms selected from group consisting of Bayesian optimization, gradient descent, and evolutionary algorithms.
6 . The method of claim 1 , further comprising:
validating the composite score against a set of standalone metrics; and
iteratively refining the evaluation framework in a feedback loop.
7 . A system for improving a performance of a large language model with respect to a set of user-defined objectives, the system comprising:
a computer processor;
memory; and
instructions stored in the memory and executable by the computer processor to cause the computer processor to perform operations, the operations comprising:
receiving the set of user-defined objectives for evaluating model responses generated by the large language model, wherein the set of user-defined objectives include at least one from semantic accuracy, grammatical soundness, and expert alignment;
based on the user-defined objectives, selecting a set of evaluation metrics from a predefined library;
determining, using the selected set of evaluation metrics, evaluation scores for model responses generated by a large language model, wherein the evaluation scores include a semantic score and a lexical score;
calculating a composite score from the evaluation scores, wherein the composite score is adjusted for divergences between the semantic score and the lexical score, wherein the composite score is calculated as:
S
comp
=
(
α
*
(
S
lex
+
b
lex
)
p
lex
+
(
1
-
α
)
*
α
*
(
S
sem
+
b
sem
)
p
sem
)
1
P
1
+
λ
×
penalty
wherein:
S comp is the composite similarity;
α is a weighted hyperparameter;
S lex is a lexical similarity;
b lex is a lexical bias;
p lex is a lexical tuning hyperparameter;
S sem is the semantic similarity;
b sem is a semantic bias;
p sem is a semantic tuning hyperparameter:
p is a mean behavior scaling factor;
λ is a penalty scaling factor; and
penalty is corrective factor for S lex -S sem divergence; and
tuning a set of hyperparameters of the large language models based on the composite score, wherein the tuning customizes the large language model according to the user-defined objectives by aligning the performance of the large language model according to the user-defined objectives.
8 . The system of claim 7 , wherein the operations further comprise:
collecting a set of evaluation data, wherein the evaluation data includes responses from subject matter experts and the model responses generated by the large language models;
preprocessing the collected evaluation data, wherein the preprocessing includes one or more natural language processing techniques to normalize and tokenize the evaluation data.
9 . The system of claim 7 , wherein the semantic score is calculated from evaluation metrics selected from a ROGUE score, a BLEU score, and combinations thereof.
10 . The system of claim 7 , wherein the lexical score is calculated from evaluation metrics selected from a BERT-Score, a BART-Score, a METEOR score, and combinations thereof.
11 . The system of claim 7 , wherein tuning the set of hyperparameters further comprises:
applying one or more optimization algorithms selected from group consisting of Bayesian optimization, gradient descent, and evolutionary algorithms.
12 . The system of claim 7 , wherein the operations further comprise:
validating the composite score against a set of standalone metrics; and
iteratively refining the evaluation framework in a feedback loop.
13 . A computer program product for improving a performance of a large language model with respect to a set of user-defined objectives, the computer program product comprising a non-transitory computer-readable medium storing program code that, when executed by a computer processor of a computing system, causes the computing system to perform operations comprising:
receiving the set of user-defined objectives for evaluating model responses generated by the large language model, wherein the set of user-defined objectives include at least one from semantic accuracy, grammatical soundness, and expert alignment;
based on the user-defined objectives, selecting a set of evaluation metrics from a predefined library;
determining, using the selected set of evaluation metrics, evaluation scores for model responses generated by a large language model, wherein the evaluation scores include a semantic score and a lexical score;
calculating a composite score from the evaluation scores, wherein the composite score is adjusted for divergences between the semantic score and the lexical score, wherein the composite score is calculated as:
S
comp
=
(
α
*
(
S
lex
+
b
lex
)
p
lex
+
(
1
-
α
)
*
α
*
(
S
sem
+
b
sem
)
p
sem
)
1
P
1
+
λ
×
penalty
wherein:
S comp is the composite similarity;
α is a weighted hyperparameter;
S lex is a lexical similarity;
b lex is a lexical bias;
p lex is a lexical tuning hyperparameter;
S sem is the semantic similarity;
b sem is a semantic bias;
p sem is a semantic tuning hyperparameter;
p is a mean behavior scaling factor;
λ is a penalty scaling factor; and
penalty is corrective factor for S lex -S sem divergence; and
tuning a set of hyperparameters of the large language models based on the composite score, wherein the tuning customizes the large language model according to the user-defined objectives by aligning the performance of the large language model according to the user-defined objectives.
14 . The computer program product of claim 13 , wherein the computing system further performs the operations of:
collecting a set of evaluation data, wherein the evaluation data includes responses from subject matter experts and the model responses generated by the large language models;
preprocessing the collected evaluation data, wherein the preprocessing includes one or more natural language processing techniques to normalize and tokenize the evaluation data.
15 . The computer program product of claim 13 , wherein the semantic score is calculated from evaluation metrics selected from a ROGUE score, a BLEU score, and combinations thereof.
16 . The computer program product of claim 13 , wherein the lexical score is calculated from evaluation metrics selected from a BERT-Score, a BART-Score, a METEOR score, and combinations thereof.
17 . The computer program product of claim 13 , wherein tuning the set of hyperparameters further comprises:
applying one or more optimization algorithms selected from group consisting of Bayesian optimization, gradient descent, and evolutionary algorithms.