Monitoring compliance of a generative language model with an output characteristic rubric

1 . A computing system for monitoring language model compliance with a rubric of one or more output characteristics, the computing system comprising:

processing circuitry configured to:

interface with a trained generative model that receives input of a prompt including natural language text input and, in response, generates an output that includes natural language text output;

monitor compliance of the generative model with the rubric, by feeding the output of the generative model to a rubric classifier that is in response configured to generate a predicted classification for an output characteristic in the rubric; and

output the predicted classification, wherein

the generative model is a generative language model;

the generative language model is intermittently updated over a time period; and

the processing circuitry is configured to monitor compliance by feeding a plurality of outputs of the generative language model to the rubric classifier at a series of points in time during the time period, to thereby generate a time-series of predicted classifications for the output characteristic in the rubric.

2 . The computing system of claim 1 , wherein

the generative language model is a first generative language model;

the computing system further comprises a second generative language model; and

the plurality of outputs are from synthetic interaction sessions between the first generative language model and the second generative language model.

3 . The computing system of claim 1 , wherein

the output characteristic is one of a plurality of output characteristics included in the rubric of the generative model; and

the rubric classifier is configured to, in response to being fed the output of the generative model, generate the predicted classification for each of the plurality of output characteristics in the rubric.

4 . The computing system of claim 3 , wherein

the rubric classifier includes a plurality of classifier models; and

each of the plurality of classifier models has been trained to receive natural language text and generate a predicted classification for a respective one of the plurality of output characteristics, respectively, based on ground truth data.

5 . The computing system of claim 4 , wherein the ground truth data is a rubric database comprising outputs of a plurality of generative language models with corresponding classifications for each of the plurality of output characteristics.

6 . The computing system of claim 1 , wherein the at least one processor is further configured to take an action based upon the predicted classification for the characteristic.

7 . The computing system of claim 6 , wherein the action is to generate a subsequent prompt inputted into the generative model to thereby improve the predicted classification for the characteristic.

8 . The computing system of claim 7 , wherein the subsequent prompt is generated based on a calculated difference between the predicted classification and a target classification.

9 . A method for monitoring language model compliance with a rubric of output characteristics, the method comprising, via processing circuitry of a computing system:

interfacing with a trained generative model that receives input of a prompt including natural language text input and, in response, generates an output that includes natural language text output;

monitoring compliance of the generative model with the rubric, by feeding the output of the generative model to a rubric classifier configured to generate a predicted classification for an output characteristic in the rubric; and

outputting the predicted classification, wherein

the generative model is a generative language model;

the generative language model is intermittently updated over a time period; and

compliance is monitored by feeding a plurality of outputs of the generative language model to the rubric classifier at a series of points in time during the time period, to thereby generate a time-series of predicted classifications for the output characteristic in the rubric.

10 . The method of claim 9 , wherein the predicted classification is a numerical classification or a qualitative classification.

11 . The computing system of claim 3 , wherein at least one of the plurality of output characteristics comprises cultural sensitivity, fairness, inclusivity, wittiness, assertiveness, patience, friendliness, formality, empathy, profanity, verbosity, tone, topical relevance, factuality, and/or creativity.

12 . The method of claim 9 , wherein

the generative language model is a first generative language model; and

the plurality of outputs are from synthetic interaction sessions between the first generative language model and a second generative language model.

13 . The method of claim 9 , wherein

the output characteristic is one of a plurality of output characteristics included in the rubric of the generative model; and

the rubric classifier is configured to generate a predicted classification for each of the plurality of output characteristics in the rubric.

14 . The method of claim 13 , wherein

the rubric classifier includes a plurality of classifier models; and

each of the plurality of classifier models has been trained to receive natural language text and generate a predicted classification for a respective one of the plurality of output characteristics, respectively, based on ground truth data.

15 . The method of claim 14 , wherein the ground truth data is stored in a rubric database comprising outputs of a plurality of generative language models with corresponding classifications for each of one or more characteristics.

16 . The method of claim 9 , wherein

the method further includes taking an action based upon the predicted classification for the output characteristic.

17 . The method of claim 16 , wherein

the action is to generate a subsequent prompt inputted into the generative model to thereby improve the predicted classification for the output characteristic; and

the subsequent prompt is generated based on a calculated difference between the predicted classification and a target classification.

18 . A computing system for monitoring language model compliance with a rubric of one or more output characteristics, the computing system comprising:

processing circuitry configured to:

cause a interaction interface for a trained generative model to be presented;

receive, via the interaction interface, natural language text input from a user for the trained generative model to generate an output;

generate a prompt based at least on the natural language text input from the user;

provide the prompt to the trained generative model;

generate, in response to the prompt, the output via the trained generative model;

based on the output of the generative model, generating a predicted classification for an output characteristic in the rubric; and

outputting the predicted classification, wherein

the generative model is a generative language model;

the generative language model is intermittently updated over a time period; and

compliance is monitored by feeding a plurality of outputs of the generative language model to a rubric classifier at a series of points in time during the time period, to thereby generate a time-series of predicted classifications for the output characteristic in the rubric.

19 . The computing system of claim 18 , wherein the generative model is a multi-modal generative model configured to receive multi-modal input.

20 . The method of claim 13 , wherein at least one of the plurality of output characteristics comprises cultural sensitivity, fairness, inclusivity, wittiness, assertiveness, patience, friendliness, formality, empathy, profanity, verbosity, tone, topical relevance, factuality, and/or creativity.