SELF-TESTING A VIRTUAL AI REPRESENTATIVE
Disclosed are approaches for testing virtual artificially intelligent (AI) agents. In some examples, user inputs are generated automatically across a variety of contexts. The automatically generated user inputs are sent to an AI agent and analytically analyzed to assess coherency and relevance. A self-test report of the AI agent can then be generated based on the assessed coherency and relevance.
1 . A method for testing a virtual artificially intelligent (AI) agent comprising the steps of:
automatically generating simulated user inputs across a plurality of defined conversational contexts;
transmitting the simulated one or more user inputs to the AI agent;
processing the simulated user inputs sent to the AI agent;
recording the AI agent's responses to the simulated inputs; analyzing the recorded responses to determine at least one of conversational coherency and contextual relevance based on predefined evaluation metrics; and
generating a self-test report of the AI agent based on the assessed coherency and relevance;
2 . The method of claim 1 , further comprising the step of performing integrity checks on a knowledge base associated with the AI agent to verify the accuracy and currency of stored information.
3 . The method of claim 1 , further comprising the steps of:
generating one or more stress tests including edge cases and ambiguous queries; and
analyzing the one or more AI agent's responses to the stress tests to determine robustness.
4 . The method of claim 1 , further comprising the steps of:
assessing a Large Language Models (LLM) conversational responses; and
evaluating at least one visual or auditory output generated by the AI agent's action controller and state manager unit for consistency with expected response templates
5 . The method of claim 1 , wherein a fake user unit is loaded with profile data selected to influence the AI agent's response style.
6 . The method of claim 5 , wherein the fake user unit is loaded with profile data selected to influence the AI agent's response style and content.
7 . The method of claim 5 , further comprising the steps of:
receiving by the fake user input unit a text message initiated by an AI representative; and
processing the text message by the fake user input unit to produce a relevant and coherent reply.
8 . The method of claim 5 , further comprising simulating a conversation loop wherein the AI agent and the fake user exchange text messages until all defined conversation states are reached or a human operator intervenes.
9 . The method of claim 6 , further comprising the step of:
exchanging messages iteratively between the AI representative and the fake user input unit until an AI representative's State Manager Unit has explored all states or a human intervenes.
10 . The method of claim 1 , further comprising detecting a predefined user signal or phrase; and
transferring control of the session from the AI agent to a human operator upon detection of the signal.
11 . The methos of claim 1 , further comprising the step of recording processing of the one or more sent inputs to the AI agent.
12 . An information handling system for initiating a human takeover by a virtual artificially intelligent (AI) agent artificially intelligent (AI) system, comprising:
a plurality of processors;
a memory coupled to at least one of the processors;
a set of computer program instructions stored in the memory and executed by at least one of the processors in order to perform the steps of:
generating automatically user inputs across a variety of contexts;
sending the automatically generated user inputs to the AI agent;
recording processing of the sent input to the AI agent;
analytically analyzing the recorded processing to assess coherency and relevance; and
generating a self-test report of the AI agent based on the assessed coherency and relevance.
13 . The information handling system of claim 12 , wherein a self-testing unit is configured to perform integrity checks on a knowledge base unit for information accuracy and currency.
14 . The information handling system of claim 12 , further comprising the steps of:
generating stress tests; and
analyzing the stress tests.
15 . The information handling system of claim 12 , further comprising the steps of:
assessing a Large Language Models (LLM) conversational responses; and
confirming an accuracy of visual and auditory outputs from an AI representative's action controller and a state manager unit.
16 . The information handling system of claim 12 , wherein a fake user unit is loaded with profile information that influences a conversation's flow and context, along with responses of an AI representative.
17 . The information handling system of claim 12 , further comprising:
receiving by a fake user input unit a text message initiated by an AI representative; and
processing the text message by the fake user input unit to produce a relevant and coherent reply.
18 . The information handling system claim 17 , further comprising:
exchanging messages between iteratively between the AI representative and the fake user input unit an AI representative's State Manager Unit has explored all states or a human intervenes.
19 . A computer program product for testing a virtual artificially intelligent (AI) agent having program instructions embodied therewith, the program instructions executable on a processing circuit to cause the processing circuit to perform the steps comprising:
generating automatically user inputs across a variety of contexts;
sending the automatically generated user inputs to the AI agent;
recording processing of the sent input to the AI agent;
analytically analyzing the recorded processing to assess coherency and relevance; and
generating a self-test report of the AI agent based on the assessed coherency and relevance.
20 . The computer program product of claim 19 , wherein a self-testing unit is configured to perform integrity checks on a knowledge base unit for information accuracy and currency.
21 . The computer program product of claim 19 , further comprising:
generating stress tests; and
analyzing the stress tests.
22 . The computer program product of claim 19 , further comprising:
assessing a Large Language Models (LLM) conversational responses; and
confirming an accuracy of visual and auditory outputs from an AI representative's action controller and a state manager unit.
23 . The computer program product of claim 19 , wherein a fake user unit is loaded with profile information that influences a conversation's flow and context, along with responses of an AI representative.
24 . The computer program product of claim 19 , further comprising:
receiving by a fake user input unit a text message initiated by an AI representative; and
processing the text message by the fake user input unit to produce a relevant and coherent reply.