Granted Patent B2

US 12,737,537 · App. 18/585,145 · Granted Sep 15, 2026

Inventors: Warren Nicholas Mante (Sachse, TX); Warren Thomas Bloomer Lucas (Middletown, CT); Sourav Mazumder (Contra Costa, CA); Andrew R. Freed (Cary, NC); Stefan A. G. Van Der Stockt (Austin, TX)

Assignee: International Business Machines Corporation

G06F40/20G06Q30/018

Abstract

Claims (83)

1 . A computer-implemented method for governing responses generated by a language large learning model (LLM), the computer-implemented method comprising:

setting up an initial risk assessment framework for a model of an LLM in a question repository, wherein setting up further comprises:

defining a reference set of inputs and output pairs for the LLM wherein the reference set of inputs and output pairs are actual inputs and reference outputs;

defining a set of metadata associated with the reference set of inputs and output pairs, wherein the set of metadata including at least prioritization, sensitive-content designation, and topic area;

assigning the set of metadata to each pair of the reference set of inputs and output pairs;

defining a set of evaluation criteria including

(i) an exact match criterion computed using a normalized Damerau-Levenshtein score, (ii) a semantic match criterion computed using a cosine similarity score, and

(iii) a prohibited word or phrase criterion determined by detecting presence of a specified word and/or phrase; assigning the set of evaluation criteria to an organizational risk framework;

associating the set of metadata to the set of evaluation criteria;

evaluating the initial risk assessment framework and the model of the LLM after a predetermined interval, further comprising:

passing all inputs from the reference set of inputs and output pairs to the LLM, and collecting all outputs from the reference set of inputs and output pairs at a predetermined interval;

evaluating and producing an evaluation metric of all the inputs and all of the outputs based on one or more policies, further comprising:

generating new outputs and comparing the new outputs against the reference set of inputs and output pairs; and

generating scores for the new outputs and adding the scores to a historical data for future analysis;

summarizing results of the evaluation metric across all of the reference set of inputs and output pairs; and

notifying one or more users when the results of the evaluation metric exceed a predetermined metric threshold;

responsive to determining that the evaluation metric exceeds the predetermined metric threshold, automatically initiating, in a governance risk and compliance (GRC) system, a workflow that includes the actual input, the reference output, actual output and confidence score for review; and

saving data associated with the result of the evaluation metric and all of the reference set of inputs and output pairs to a data repository for trend analysis.

2 . The computer-implemented method of claim 1 , wherein the predetermined interval comprises every hour or every week.

3 . The computer-implemented method of claim 1 , wherein the set of evaluation criteria further comprises:

the exact match criterion in which the normalized Damerau-Levenshtein score is required to equal 1.0 to pass;

ii) the semantic match criterion in which the cosine similarity score is required to exceed a threshold X to pass; and

(iii) the prohibited word or phrase criterion in which the presence of the specified word and/or phrase triggers a fail.

4 . The computer-implemented method of claim 1 , wherein assigning the set of metadata to each pair of the reference set of inputs and output pairs further comprises assessing the set of metadata using a rules engine or classifier including at least one of: keyword evaluation to mark an input/output pair as sensitive content; and a topic classifier to assign the topic area.

5 . The computer-implemented method of claim 1 , wherein the set of metadata comprises, prioritization, sensitive content and topic areas.

6 . The computer-implemented method of claim 1 , wherein automatically initiating the workflow further comprises routing the workflow to at least one user selected from an administrator, a data scientist, risk management personnel, and an internal auditor, and wherein the workflow is reviewed to determine whether the actual output is within specification under the evaluation criteria or whether the model of the LLM is to be fine-tuned.

7 . The computer-implemented method of claim 1 , wherein associating the set of metadata to the set of evaluation criteria can include associating high priority to the exact match criterion; associating low priority and low sensitivity content to the semantic match criterion; and associating sensitive content to the prohibited word or phrase criterion.

8 . A computer program product for governing responses generated by a language large learning model (LLM), the computer program product comprising:

one or more computer readable storage media and program instructions stored on the one or more computer readable storage media, the program instructions comprising the steps of:

setting up an initial risk assessment framework for a model of an LLM in a question repository, wherein setting up further comprises:

defining a reference set of inputs and output pairs for the LLM wherein the reference set of inputs and output pairs are actual inputs and reference outputs;

defining a set of metadata associated with the reference set of inputs and output pairs, wherein the set of metadata including at least prioritization, sensitive-content designation, and topic area;

assigning the set of metadata to each pair of the reference set of inputs and output pairs;

defining a set of evaluation criteria including (i) an exact match criterion computed using a normalized Damerau-Levenshtein score, (ii) a semantic match criterion computed using a cosine similarity score, and

(iii) a prohibited word or phrase criterion determined by detecting presence of a specified word and/or phrase;

assigning the set of evaluation criteria to an organizational risk framework;

and

associating the set of metadata to the set of evaluation criteria; evaluating the initial risk assessment framework and the model of the LLM after a predetermined interval, further comprising:

passing all inputs from the reference set of inputs and output pairs to the LLM, and collecting all outputs from the reference set of inputs and output pairs at a predetermined interval;

evaluating and producing an evaluation metric of all the inputs and all of the outputs based on one or more policies, further comprising:

generating new outputs and comparing the new outputs against the reference set of inputs and output pairs; and

generating scores for the new outputs and adding the scores to a historical data for future analysis;

summarizing results of the evaluation metric across all of the reference set of inputs and output pairs; and

notifying one or more users when the results of the evaluation metric exceed a predetermined metric threshold;

responsive to determining that the evaluation metric exceeds the predetermined metric threshold, automatically initiating, in a governance risk and compliance (GRC) system, a workflow that includes the actual input, the reference output, actual output and confidence score for review; and

saving data associated with the result of the evaluation metric and all of the reference set of inputs and output pairs to a data repository for trend analysis.

9 . The computer program product of claim 8 , wherein the predetermined interval comprises every hour or every week.

10 . The computer program product of claim 8 , wherein the set of evaluation criteria further comprises (i) the exact match criterion in which the normalized Damerau-Levenshtein score is required to equal 1.0 to pass;

(ii) the semantic match criterion in which the cosine similarity score is required to exceed a threshold X to pass; and

iii) the prohibited word or phrase criterion in which the presence of the specified word and/or phrase triggers a fail.

11 . The computer program product of claim 8 , wherein assigning the set of metadata to each pair of the reference set of inputs and output pairs further comprises:

assessing the set of metadata using a rules engine or classifier including at least one of: keyword evaluation to mark an input/output pair as sensitive content; and a topic classifier to assign the topic area.

12 . The computer program product of claim 8 , wherein the set of metadata comprises, prioritization, sensitive content and topic areas.

13 . The computer program product of claim 8 , wherein automatically initiating the workflow further comprises routing the workflow to at least one user selected from an administrator, a data scientist, risk management personnel, and an internal auditor, and wherein the workflow is reviewed to determine whether the actual output is within specification under the evaluation criteria or whether the model of the LLM is to be fine-tuned.

14 . The computer program product of claim 8 , wherein associating the set of metadata to the set of evaluation criteria can include, a high priority is associated to exact match, a low priority and low sensitivity content are required a semantic match and sensitive contents must not contain a certain letter and/or phrase.

15 . A computer system for governing responses generated by a language learning model (LLM), the computer system comprising:

one or more computer processors;

one or more computer readable storage media; and

program instructions stored on the one or more computer readable storage media for execution by at least one of the one or more computer processors, the program instructions comprising the steps of:

setting up an initial risk assessment framework for a model of an LLM in a question repository, wherein setting up further comprises:

defining a reference set of inputs and output pairs for the LLM wherein the reference set of inputs and output pairs are actual inputs and reference outputs; defining a set of metadata associated with the reference set of inputs and output pairs, wherein the set of metadata including at least prioritization, sensitive-content designation, and topic area;

assigning the set of metadata to each pair of the reference set of inputs and output pairs;

defining a set of evaluation criteria including

(i) an exact match criterion computed using a normalized Damerau-Levenshtein score, (ii) a semantic match criterion computed using a cosine similarity score, and

(iii) a prohibited word or phrase criterion determined by detecting presence of a specified word and/or phrase; assigning the set of evaluation criteria to an organizational risk framework;

and

associating the set of metadata to the set of evaluation criteria; evaluating the initial risk assessment framework and the model of the LLM after a predetermined interval, further comprising:

passing all inputs from the reference set of inputs and output pairs to the LLM, and collecting all outputs from the reference set of inputs and output pairs at a predetermined interval;

evaluating and producing an evaluation metric of all the inputs and all of the outputs based on one or more policies, further comprising:

generating new outputs and comparing the new outputs against the reference set of inputs and output pairs; and

generating scores for the new outputs and adding the scores to a historical data for future analysis;

summarizing results of the evaluation metric across all of the reference set of inputs and output pairs; and

notifying one or more users when the results of the evaluation metric exceed a predetermined metric threshold; responsive to determining that the evaluation metric exceeds the predetermined metric threshold, automatically initiating, in a governance risk and compliance (GRC) system, a workflow that includes the actual input, the reference output, actual output and confidence score for review; and

saving data associated with the result of the evaluation metric and all of the reference set of inputs and output pairs to a data repository for trend analysis.

16 . The computer system of claim 15 , wherein the predetermined interval comprises every hour or every week.

17 . The computer system of claim 15 , wherein the set of evaluation criteria further comprises (i) the exact match criterion in which the normalized Damerau-Levenshtein score is required to equal 1.0 to pass;

(ii) the semantic match criterion in which the cosine similarity score is required to exceed a threshold X to pass; and

(iii) the prohibited word or phrase criterion in which the presence of the specified word and/or phrase triggers a fail.

18 . The computer system of claim 15 , wherein assigning the set of metadata to each pair of the reference set of inputs and output pairs further comprises:

assessing the set of metadata using a rules engine or classifier including at least one of:

keyword evaluation to mark an input/output pair as sensitive content; and a topic classifier to assign the topic area.

19 . The computer system of claim 15 , wherein the set of metadata comprises, prioritization, sensitive content and topic areas.

20 . The computer system of claim 15 , wherein automatically initiating the workflow further comprises routing the workflow to at least one user selected from an administrator, a data scientist, risk management personnel, and an internal auditor, and wherein the workflow is reviewed to determine whether the actual output is within specification under the evaluation criteria or whether the model of the LLM is to be fine-tuned.

Assignments (1)

Continuity (1)

References Cited (19)