Granted Patent B2

US 12,299,140 · App. 18/947,102 · Granted May 13, 2025

Inventors: Sofia Rahman (New York, NY); James Myers (New York, NY); Prashant Praveen (New York, NY); Shardul Malviya (London, GB); Wayne Liao (London, GB); Deepak Jain (London, GB); Samantha Cory (London, GB); Mariusz Saternus (Cracow, PL); Daniel Lewandowski (Cracow, PL); Biraj Krushna Rath (London, GB); Stuart Murray (London, GB); Philip Davies (London, GB); Payal Jain (London, GB); Tariq Husayn Maonah (London, GB); Vishal Mysore (Mississauga, CA); Ramkumar Ayyadurai (Jersey City, NJ); Chamindra Desilva (London, GB); William Franklin Cameron (Jacksonville, FL); Miriam Silver (Tel Aviv, IL); Prithvi Narayana Rao (Allen, TX); Pramod Goyal (Ahmedabad, IN); Manjit Rajaretnam (Irving, TX)

Assignee: Citibank, N.A.

G06F21/577G06F21/552

Abstract

Claims (77)

1. A non-transitory computer-readable storage medium comprising instructions thereon, wherein the instructions when executed by at least one data processor of a system, cause the system to:

receive, from a computing device, an output generation request comprising a prompt for generation of an output using a multi-model superstructure, the multi-model superstructure comprising: (i) a first set of AI models and (ii) a second set of AI models;

supply the output generation request to one or more AI models of the first set of AI models to generate a set of model-specific outputs;

dynamically route, by the multi-model superstructure, the set of model-specific outputs of the first set of AI models to one or more AI models of the second set of AI models by:

determining a set of dimensions of the set of model-specific outputs against which to evaluate the set of model-specific outputs, and

for each particular dimension in the determined set of dimensions, identifying, by the multi-model superstructure, the one or more AI models of the second set of AI models used to test the particular dimension;

for each particular dimension in the determined set of dimensions, evaluate, by the second set of AI models, each particular model-specific output of the set of model-specific outputs against a set of assessments to determine satisfaction of the particular model-specific output with a corresponding set of assessment metrics of each assessment by:

constructing the set of assessments including a set of seed assessments testing the particular dimension of the particular model-specific output against threshold values of the corresponding set of assessment metrics,

comparing values of the corresponding set of assessment metrics of the particular model-specific output with the threshold values of the corresponding set of assessment metrics,

using the comparison, generating a set of seed assessment results indicating a degree of satisfaction of the particular model-specific output with the threshold values of the corresponding set of assessment metrics of the set of seed assessments,

using the set of seed assessment results, dynamically constructing a set of subsequent assessments within the set of assessments constructed subsequent to the set of seed assessments, and

applying the set of subsequent assessments of the set of assessments to the particular model-specific output to generate a set of overall assessment results based on a degree of satisfaction of the particular model-specific output with the threshold values of the assessment metrics of: (i) the set of seed assessments and (ii) the set of subsequent assessments;

responsive to the set of assessment results of a particular model-specific output failing to satisfy one or more threshold values of the corresponding set of assessment metrics of the set of assessments, generate, by the second set of AI models, a set of actions to add a set of pre-loaded query context to the output generation request indicated by the particular assessment metrics; and

for each model-specific output, cause the computing device to display a graphical layout indicating the set of assessment results,

wherein the graphical layout includes (1) a first representation of the particular model-specific output and (2) a second representation of a corresponding set of actions generated.

2. The non-transitory, computer-readable storage medium of claim 1 , wherein the instructions further cause the system to:

using the generated set of actions, update the output generation request by automatically triggering an automated workflow indicated by the generated set of actions, wherein the automated workflow includes executing the generated set of actions;

using the updated output generation request, supply the updated output generation request to the one or more AI models of the first set of AI models to generate a set of updated model-specific outputs; and

evaluate, by the second set of AI models, each particular updated model-specific output of the updated model-specific outputs against the set of assessments to determine satisfaction of the particular updated model-specific output with the corresponding set of assessment metrics of each assessment.

3. The non-transitory, computer-readable storage medium of claim 1 , wherein the set of model-specific outputs is a first set of model-specific outputs, wherein the instructions further cause the system to:

provide the output generation request loaded with the pre-loaded query context to the one or more AI models of the first set of AI models to generate a second set of model-specific outputs; and

responsive to the second set of model-specific outputs satisfying each assessment metrics of the set of assessments, automatically transmit, to the computing device, the second set of model-specific outputs.

4. The non-transitory, computer-readable storage medium of claim 1 , wherein the one or more AI models of the second set of AI models are determined randomly.

5. The non-transitory, computer-readable storage medium of claim 1 , wherein the instructions further cause the system to:

establish a predefined schedule to rotate the one or more AI models in the second set of AI models,

wherein the predefined schedule is established using one or more of: (1) time intervals or (2) a number of output generation requests processed; and

using the predefined schedule, determine the one or more AI models of the second set of AI models.

6. The non-transitory, computer-readable storage medium of claim 1 ,

wherein the first set of AI models and the second set of AI models include one or more of: 1) general-purpose AI models or 2) domain-specific AI models,

wherein the set of model-specific outputs are routed to the one or more AI models of the second set of AI models trained on data sharing a common domain with one or more model-specific outputs of the set of model-specific outputs, and

wherein the domain indicates one or more of: 1) an area of knowledge, 2) a data type, 3) a guideline type, or 4) a type of task.

7. The non-transitory, computer-readable storage medium of claim 1 , wherein the instructions further cause the system to:

determine whether the particular model-specific output fails to satisfy one or more particular assessment metrics of the set of assessments using a majority vote between the one or more AI models of the second set of AI models.

8. A computing system for dynamic multi-model monitoring and validation of a generative artificial intelligence model, the system comprising:

a first set of AI models configured to generate a set of artifacts;

a second set of AI models configured to receive, from a computing device, the set of artifacts as input to generate, using at least one hardware processor of the computing device, a set of assessment results indicating a degree of satisfaction of a set of assessment metric values of the set of artifacts with a set of threshold metric values of a corresponding set of assessment metrics; and

a third set of AI models configured to:

dynamically route the set of artifacts generated by the first set of AI models to one or more AI models in the second set of AI models by:

determining a set of dimensions of the set of artifacts against which to evaluate the set of artifacts,

for each particular dimension in the determined set of dimensions, identifying the one or more AI models of the second set of AI models used to test the particular dimension, and

responsive to the set of assessment results generated by the second set of AI models failing to satisfy one or more threshold metric values of the corresponding set of assessment metrics, cause the second set of AI models to generate a set of actions to modify one or more of: 1) parameters of the first set of AI models indicated by the particular assessment metrics or 2) an output generation request configured to generate the set of artifacts using the first set of AI models.

9. The system of claim 8 , wherein the third set of AI models is further configured to:

dynamically select the one or more AI models of the second set of AI models using dimension-specific features of the particular dimension being evaluated.

10. The system of claim 8 , wherein the third set of AI models is further configured to:

for each particular artifact of the set of artifacts, cause the computing device to display a layout indicating the set of assessment results,

wherein the layout includes (1) a first representation of the particular artifact and (2) a second representation of a corresponding set of actions generated.

11. The system of claim 8 , wherein the third set of AI models is further configured to:

use the generated set of actions to update the set of artifacts by automatically triggering an automated workflow indicated by the generated set of actions,

wherein the automated workflow includes executing the generated set of actions.

12. The system of claim 8 , wherein the set of artifacts is a first set of artifacts, wherein the third set of AI models is further configured to:

provide the first set of AI models with a pre-loaded query context to generate a second set of artifacts using the first set of AI models.

13. The system of claim 8 , wherein the third set of AI models determines the one or more AI models of the second set of AI models randomly.

14. The system of claim 8 , wherein the third set of AI models establishes a predefined schedule to rotate the one or more AI models in the second set of AI models using one or more of: (1) time intervals or (2) a number of output generation requests processed.

15. A computer-implemented method for dynamic multi-model monitoring and validation of a generative artificial intelligence model, the method comprising:

receiving, from a computing device, a set of artifacts generated using a first set of AI models within a multi-model superstructure,

wherein the multi-model superstructure comprises: (i) the first set of AI models and (ii) a second set of AI models to test the first set of AI models;

dynamically routing, by the multi-model superstructure, the set of artifacts of the first set of AI models to one or more AI models of the second set of AI models by:

determining a set of dimensions of the set of artifacts against which to evaluate the set of artifacts, and

for each particular dimension in the determined set of dimensions, identifying, by the multi-model superstructure, the one or more AI models of the second set of AI models used to test the particular dimension;

for each particular dimension in the determined set of dimensions, inputting each particular artifact of the set of artifacts into the second set of AI models to use at least one hardware processor of the computing device to output an indication of satisfaction of the particular artifact with a set of assessment metrics of a set of assessments;

responsive to the particular artifact failing to satisfy one or more assessment metrics of the set of assessments, generating, by the second set of AI models, a set of actions to modify one or more of: 1) parameters of the first set of AI models indicated by the one or more assessment metrics or 2) an output generation request configured to generate the set of artifacts using the first set of AI models; and

for each model-specific output, cause the computing device to display a layout including (1) a first representation of the particular model-specific output and (2) a second representation of a corresponding set of actions generated.

16. The method of claim 15 , further comprising:

using the generated set of actions, updating the set of artifacts by automatically triggering an automated workflow indicated by the generated set of actions, wherein the automated workflow includes executing the generated set of actions; and

evaluating, by the second set of AI models, each particular updated artifact of the updated artifacts against the set of assessments to determine satisfaction of the particular updated artifact with the corresponding set of assessment metrics of each assessment.

17. The method of claim 15 , wherein the set of artifacts is a first set of artifacts, further comprising:

providing the first set of AI models with a pre-loaded query context to generate a second set of artifacts using the first set of AI models; and

responsive to each artifact in the second set of artifacts satisfying each assessment metric of the set of assessments, automatically transmitting, to the computing device, the second set of artifacts.

18. The method of claim 15 , wherein the one or more AI models of the second set of AI models are determined randomly.

19. The method of claim 15 , further comprising:

establishing a predefined schedule to rotate the one or more AI models in the second set of AI models,

wherein the predefined schedule is established using one or more of: (1) time intervals or (2) a number of output generation requests processed; and

using the predefined schedule, determining the one or more AI models of the second set of AI models.

20. The method of claim 15 ,

wherein the first set of AI models and the second set of AI models include one or more of: 1) general-purpose AI models or 2) domain-specific AI models, and

wherein the set of artifacts are routed to the one or more AI models of the second set of AI models trained on data sharing a common domain with one or more artifacts of the set of artifacts,

wherein the common domain indicates one or more of: 1) an area of knowledge, 2) a data type, 3) A guideline type, or 4) a type of task.

Assignments (1)

Continuity (14)

References Cited (154)

Cited By (1)

US 12,488,024