Confidence level for agent-generated search analytics
The technology disclosed relates to a method for inducing an agent-under-test (AUT) to disclose respective outputs in response to processing a target input probe. An agent sampling logic is configured to analyze respective outputs to determine whether the outputs represent a sample for the target input probe and to also determine a confidence level of the respective outputs being the sample for the target input probe.
1 . A computer-implemented method, including:
inducing an agent-under-test (AUT) to disclose respective outputs in response to processing a target input probe, wherein the AUT comprises an encoder and a decoder, wherein the encoder receives the target input probe and encodes the target input probe into a context vector, and wherein the decoder, conditioned on the context vector generated by the encoder, generates the respective outputs;
analyzing the respective outputs, generated by the decoder conditioned on the context vector, to determine whether the respective outputs represent a sample for the target input probe, and to determine a confidence level of the respective outputs being the sample for the target input probe; and
analyzing the respective outputs, generated by the decoder conditioned on the context vector, to generate analytics for the sample.
2 . The computer-implemented method of claim 1 , wherein the AUT is an artificial intelligence (AI) system.
3 . The computer-implemented method of claim 2 , wherein the AUT is a large language model (LLM).
4 . The computer-implemented method of claim 1 , wherein the confidence level reflects a reliability index indicating distributional completeness of the respective outputs through one or more statistical models.
5 . The computer-implemented method of claim 1 , further including inducing the AUT at periodic intervals.
6 . The computer-implemented method of claim 5 , further including for each of the periodic intervals, determining whether the respective outputs represent the sample for the target input probe, and/or determining the confidence level of the respective outputs representing the sample for the target input probe.
7 . The computer-implemented method of claim 5 , wherein the periodic intervals are second-wise.
8 . The computer-implemented method of claim 5 , wherein the periodic intervals are minute-wise, hour-wise, day-wise, week-wise, month-wise, and/or year-wise.
9 . The computer-implemented method of claim 5 , wherein the periodic intervals are retrospective and apply to the respective outputs disclosed in prior time periods.
10 . The computer-implemented method of claim 1 , wherein an inducing agent sends the target input probe to the AUT.
11 . The computer-implemented method of claim 1 , wherein the target input probe is a prompt to the AUT.
12 . The computer-implemented method of claim 11 , wherein the respective outputs are answers to the target input probe.
13 . The computer-implemented method of claim 1 , wherein the analyzing the respective outputs to generate the analytics for the sample further includes determining a share of voice percentage of features associated with the target input probe, a share of voice percentile of the features associated with the target input probe, and a share of voice proportion of the features associated with the target input probe.
14 . The computer-implemented method of claim 1 , wherein the analyzing the respective outputs to generate the analytics for the sample further includes performing sentiment analysis, bias detection, topics identification, source identification, and/or inaccuracy detection.
15 . The computer-implemented method of claim 1 , further including displaying the respective outputs for the target input probe.
16 . The computer-implemented method of claim 1 , further including displaying the determined confidence level of the respective outputs being the sample for the target input probe.
17 . The computer-implemented method of claim 1 , further including determining the confidence level for the generated analytics.
18 . The computer-implemented method of claim 17 , further including displaying the determined confidence level for the generated analytics.