Systems and methods for testing artificial intelligence systems
The technology disclosed relates to a system for constructing a probeable output generative space of an agent-under-test (AUT) for a target input probe. An agent sampling logic is configured to induce an agent-under-test (AUT) to disclose a plurality of outputs in response to processing a target input probe. A space construction logic, having access to the plurality of outputs, is configured to construct a probeable output generative space based on the plurality of outputs. A space probing logic, having access to the probeable output generative space, is configured to probe the probeable output generative space for a query, and to make available results of the probing for further analysis.
1. A system for constructing a probeable output generative space of an agent-under-test (AUT) for a target input probe, comprising:
an agent sampling logic configured to induce an agent-under-test (AUT) to disclose a plurality of outputs in response to processing a target input probe;
a space construction logic, having access to the plurality of outputs, and configured to construct a probeable output generative space based on the plurality of outputs;
a space probing logic, having access to the probeable output generative space, and configured to probe the probeable output generative space for a query, and to make available results of the probing for further analysis;
wherein the AUT is a large language model (LLM), wherein the LLM can be trained with minimal queries; and
wherein the agent sampling logic is further configured to induce the AUT to disclose the plurality of outputs in response to concurrently processing the target input probe over multiple parallel instances.
2. The system of claim 1 , wherein the query requires identifying a distribution pattern of features associated with the target input probe.
3. The system of claim 2 , wherein the features include mentions, in the probeable output generative space, of the target input probe.
4. The system of claim 2 , wherein the features include mentions, in the probeable output generative space, of one or more variations of the target input probe.
5. The system of claim 4 , wherein the agent sampling logic is further configured to induce the AUT to disclose the plurality of outputs in response to processing the target input probe and the variations of the target input probe.
6. The system of claim 2 , wherein the features include mentions, in the probeable output generative space, of concepts related to the target input probe.
7. The system of claim 2 , wherein the distribution pattern includes frequencies at which the features occur in the probeable output generative space.
8. The system of claim 7 , wherein the frequencies are determined by percentages.
9. The system of claim 2 , wherein the distribution pattern includes priminalities at which the features occur in the probeable output generative space.
10. The system of claim 9 , wherein the priminalities are determined by percentiles.
11. The system of claim 1 , wherein the agent sampling logic is further configured to induce the AUT to disclose the plurality of outputs in response to iteratively processing the target input probe over multiple sequential instances.
12. The system of claim 1 , wherein the agent sampling logic is further configured to terminate the inducement of the AUT when a termination condition is satisfied.
13. The system of claim 1 , wherein the query is designed prior to the inducement of the AUT and based on the target input probe.
14. The system of claim 13 , wherein the query is designed after the inducement of the AUT and based on contents of the probeable output generative space.