Granted Patent B1

US 12,731,053 · App. 19/571,342 · Granted Sep 8, 2026

Inventor: Imir Arifi (Nashville, TN)

Assignee: Citibank, N.A.

G06N5/04G06F16/3329

Abstract

Claims (85)

1 . One or more non-transitory computer-readable media storing instructions thereon for request execution by artificial intelligence agents, wherein the instructions cause one or more processors to:

receive a request comprising a prompt for generation of a response using an artificial intelligence agent deployed on a computing device;

determine, based on the request, a request context comprising a plurality of categorical parameters associated with the request, wherein the plurality of categorical parameters indicates one or more categories associated with the request, and wherein each category of the one or more categories is associated with a corresponding reasoning domain area of a plurality of reasoning domain areas;

compare, using an artificial intelligence model, a behavioral profile associated with the artificial intelligence agent with the plurality of categorical parameters associated with the request to obtain one or more deficient categories associated with the request, wherein the one or more deficient categories of the request correspond to one or more reasoning domain areas where an output of the artificial intelligence agent has been measured to have an agent output metric not satisfying a threshold;

compare a plurality of behavior profiles associated with a plurality of artificial intelligence agents with the one or more deficient categories;

determine, based on the comparing, that the one or more deficient categories are included in a first behavioral profile of a first domain-specific agent indicating that the first domain-specific agent is measured, based on an accuracy parameter meeting an accuracy threshold, to provide valid responses to the one or more deficient categories;

modify, based on a reasoning domain area associated with a deficient category of the one or more deficient categories, a first portion of the request to be executed by the first domain-specific agent, wherein the first portion of the request comprises a portion of the request related to the reasoning domain area associated with the deficient category;

transmit the first portion of the request to the first domain-specific agent to obtain a domain-specific response;

modify the request to incorporate the domain-specific response, wherein the domain-specific response replaces one or more portions of the request corresponding to the one or more deficient categories to address the one or more deficient categories;

generate a modified request comprising a modified prompt for generating the response and comprising the domain-specific response from the first domain-specific agent as a replacement for the first portion of the request; and

cause the modified request to be executed by the artificial intelligence agent deployed on the computing device, wherein execution of the modified request refines processing operations on the computing device for reducing a reasoning-error associated with the artificial intelligence agent.

2 . The one or more non-transitory computer-readable media of claim 1 , wherein the instructions for modifying the first portion of the request to be executed by the first domain-specific agent further cause the one or more processors to:

using the one or more deficient categories associated with the request, generate one or more sub-requests comprising the one or more portions of the request, with each portion corresponding to the deficient category of the one or more deficient categories, wherein a first sub-request of the one or more sub-requests comprises the first portion of the request; and

transmit the one or more sub-requests to one or more domain-specific agents, wherein a subset of one or more sub-requests is transmitted to the first domain-specific agent.

3 . The one or more non-transitory computer-readable media of claim 2 , wherein each sub-request of the one or more sub-requests is associated with corresponding metadata indicating that an associated sub-request corresponds to at least one of the one or more deficient categories, and wherein each response to the one or more sub-requests replaces a corresponding portion of the request.

4 . The one or more non-transitory computer-readable media of claim 1 , wherein the instructions further cause the one or more processors to:

receive the response to the modified request executed by the artificial intelligence agent;

determine, using a machine learning model trained to measure a degree of accuracy of agent responses to requests, the accuracy parameter corresponding to the response to the modified request; and

determine whether the accuracy parameter corresponding to the response meets the accuracy threshold.

5 . The one or more non-transitory computer-readable media of claim 1 , wherein the instructions further cause the one or more processors to:

add the request and the domain-specific response to a dataset for training the artificial intelligence agent, wherein the response is associated with the request within the dataset.

6 . The one or more non-transitory computer-readable media of claim 1 , wherein the instructions for generating the modified request further cause the one or more processors to:

based on the domain-specific response from the first domain-specific agent, generate response data corresponding to the one or more reasoning domain areas associated with the first portion of the request, wherein the response data includes structured data extracted from the domain-specific response from the first domain-specific agent.

7 . The one or more non-transitory computer-readable media of claim 1 , wherein the instructions further cause the one or more processors to:

receive the response to the modified request executed by the artificial intelligence agent;

determine, using a context similarity machine learning model, whether a response context of the response matches one or more bias contexts retrieved from a bias database, wherein the context similarity machine learning model is trained to measure a degree of context similarity between a particular bias context from the bias database and a particular response; and

based on determining that the response context matches a bias context from the bias database, generate an indication of a bias detected within the response.

8 . The one or more non-transitory computer-readable media of claim 7 , wherein the instructions further cause the one or more processors to:

receive a user behavioral profile associated with a user that generated the request;

determine whether the bias detected within the response matches a user preferred bias within the user behavioral profile;

based on determining that the bias detected within the response does not match the user preferred bias within the user behavioral profile, modify the request with a set of instructions to the artificial intelligence agent to avoid the bias, wherein the set of instructions comprise the indication of the bias; and

based on determining that the bias detected within the response matches the user preferred bias within the user behavioral profile, refrain from modifying the request.

9 . A method comprising:

receiving a request comprising a prompt for generation of a response using an artificial intelligence agent deployed on a computing device;

determining, based on the request, a request context comprising a plurality of categorical parameters associated with the request;

comparing, using an artificial intelligence model, a behavioral profile associated with the artificial intelligence agent with the plurality of categorical parameters associated with the request to obtain one or more deficient categories associated with the request that correspond to one or more reasoning domain areas where an output of the artificial intelligence agent has been measured to have an agent output metric satisfying a threshold;

comparing a plurality of behavior profiles associated with a plurality of artificial intelligence agents with the one or more deficient categories;

determining, based on the comparing, that the one or more deficient categories are included in a first behavioral profile of a first domain-specific agent indicating that the first domain-specific agent is measured, based on an accuracy parameter meeting an accuracy threshold, to provide valid responses to the one or more deficient categories;

modifying, based on a reasoning domain area associated with a deficient category of the one or more deficient categories, a first portion of the request to be executed by the first domain-specific agent;

transmitting the first portion of the request to the first domain-specific agent to obtain a domain-specific response;

modifying the request to incorporate the domain-specific response, wherein the domain-specific response replaces one or more portions of the request corresponding to the one or more deficient categories to address the one or more deficient categories;

generating a modified request comprising a modified prompt for generating the response and comprising the domain-specific response from the first domain-specific agent as a replacement for the first portion of the request; and

causing the modified request to be executed by the artificial intelligence agent deployed on the computing device, wherein execution of the modified request refines processing operations on the computing device for reducing a reasoning-error associated with the artificial intelligence agent.

10 . The method of claim 9 , wherein modifying the first portion of the request to be executed by the first domain-specific agent further comprises:

using the one or more deficient categories associated with the request, generating one or more sub-requests comprising the one or more portions of the request, with each portion corresponding to the deficient category of the one or more deficient categories, wherein a first sub-request of the one or more sub-requests comprises the first portion of the request; and

transmitting the one or more sub-requests to one or more domain-specific agents, wherein a subset of one or more sub-requests is transmitted to the first domain-specific agent.

11 . The method of claim 10 , wherein each sub-request of the one or more sub-requests is associated with corresponding metadata indicating that an associated sub-request corresponds to at least one of the one or more deficient categories, and wherein each response to the one or more sub-requests replaces a corresponding portion of the request.

12 . The method of claim 9 , further comprising:

receiving the response to the modified request executed by the artificial intelligence agent;

determining, using a machine learning model trained to measure a degree of accuracy of agent responses to requests, the accuracy parameter corresponding to the response to the modified request; and

determining whether the accuracy parameter corresponding to the response meets the accuracy threshold.

13 . The method of claim 9 , further comprising:

adding the request and the response to a dataset for training the artificial intelligence agent, wherein the response is associated with the request within the dataset.

14 . The method of claim 9 , further comprising:

based on the domain-specific response from the first domain-specific agent, generating response data corresponding to the one or more reasoning domain areas associated with the first portion of the request, wherein the response data includes structured data extracted from the domain-specific response from the first domain-specific agent.

15 . The method of claim 9 , further comprising:

receiving the response to the modified request executed by the artificial intelligence agent;

determining, using a context similarity machine learning model, whether a response context of the response matches one or more bias contexts retrieved from a bias database, wherein the context similarity machine learning model is trained to measure a degree of context similarity between a particular bias context from the bias database and a particular response; and

based on determining that the response context matches a bias context from the bias database, generating an indication of a bias detected within the response.

16 . The method of claim 15 , further comprising:

receiving a user behavioral profile associated with a user that generated the request;

determining whether the bias detected within the response matches a user preferred bias within the user behavioral profile;

based on determining that the bias detected within the response does not match the user preferred bias within the user behavioral profile, modifying the request with a set of instructions to the artificial intelligence agent to avoid the bias, wherein the set of instructions comprise the indication of the bias; and

based on determining that the bias detected within the response matches the user preferred bias within the user behavioral profile, refraining from modifying the request.

17 . A system for testing and monitoring artificial intelligence agents, the system comprising:

one or more processors; and

one or more non-transitory, computer-readable storage media storing instructions, which when executed by the one or more processors cause the one or more processors to perform operations comprising:

receiving a request comprising a prompt for generation of a response using an artificial intelligence agent deployed on a computing device;

determining, based on the request, a request context comprising a plurality of categorical parameters associated with the request, wherein the plurality of categorical parameters indicates one or more categories associated with the request, and wherein each category of the one or more categories is associated with a corresponding reasoning domain area of a plurality of reasoning domain areas;

comparing, using an artificial intelligence model, a behavioral profile associated with the artificial intelligence agent with the plurality of categorical parameters associated with the request to obtain one or more deficient categories associated with the request, wherein the one or more deficient categories of the request correspond to one or more reasoning domain areas where an output of the artificial intelligence agent has been measured to have an agent output metric satisfying a threshold;

comparing a plurality of behavior profiles associated with a plurality of artificial intelligence agents with the one or more deficient categories;

determining, based on the comparing, that the one or more deficient categories are included in a first behavioral profile of a first domain-specific agent indicating that the first domain-specific agent is measured, based on an accuracy parameter meeting an accuracy threshold, to provide valid responses to the one or more deficient categories;

modifying, based on a reasoning domain area associated with a deficient category of the one or more deficient categories, a first portion of the request to be executed by the first domain-specific agent, wherein the first portion of the request comprises a portion of the request related to the reasoning domain area associated with the deficient category;

transmitting the first portion of the request to the first domain-specific agent to obtain a domain-specific response;

modifying the request to incorporate the domain-specific response, wherein the domain-specific response replaces one or more portions of the request corresponding to the one or more deficient categories to address the one or more deficient categories;

generating a modified request comprising a modified prompt for generating the response and comprising the domain-specific response from the first domain-specific agent as a replacement for the first portion of the request; and

causing the modified request to be executed by the artificial intelligence agent deployed on the computing device, wherein execution of the modified request refines processing operations on the computing device for reducing a reasoning-error associated with the artificial intelligence agent.

18 . The system of claim 17 , wherein modifying the first portion of the request to be executed by the first domain-specific agent further comprises:

using the one or more deficient categories associated with the request, generating one or more sub-requests comprising the one or more portions of the request, with each portion corresponding to the deficient category of the one or more deficient categories, wherein a first sub-request of the one or more sub-requests comprises the first portion of the request; and

transmitting the one or more sub-requests to one or more domain-specific agents, wherein a subset of one or more sub-requests is transmitted to the first domain-specific agent.

19 . The system of claim 18 , wherein each sub-request of the one or more sub-requests is associated with corresponding metadata indicating that an associated sub-request corresponds to at least one of the one or more deficient categories, and wherein each response to the one or more sub-requests replaces a corresponding portion of the request.

20 . The system of claim 17 , further comprising:

receiving the response to the modified request executed by the artificial intelligence agent;

determining, using a machine learning model trained to measure a degree of accuracy of agent responses to requests, the accuracy parameter corresponding to the response to the modified request; and

determining whether the accuracy parameter corresponding to the response meets the accuracy threshold.

Assignments (1)

References Cited (29)