Detecting hallucination in a language model

1 . A system comprising:

at least one computer processor; and

computer storage media storing computer-useable instructions that, when used by the at least one computer processor, cause the at least one computer processor to perform operations comprising:

receiving a first output of a language model based on the language model processing a first input, wherein the first input includes: a request, target content, and an example output comprising a representation that indicates, at least partially how the first output should be formatted, and wherein the first output comprises a first output confidence and first generative text;

providing a second input to the language model, wherein the second input comprises: the target content and the first output, and wherein the second input excludes any examples;

in response to the providing the second input, receiving a second output from the language model, wherein the second output comprises a second output confidence;

comparing the first output confidence and the second output confidence;

based at least in part on the comparing, determining a first hallucination score indicating a likelihood of hallucination in the first output; and

based at least in part on the first hallucination score exceeding a threshold: refraining from presenting, at a user device, the first generative text, generating second generative text associated with a second hallucination score that does not exceed the threshold, and causing presentation, at the user device, of the second generative text.

2 . The system of claim 1 , wherein the operations further comprise:

determine a composite frequency score by summing each frequency score determined for each token; and

determine whether the composite frequency score exceeds a second threshold, wherein the determining of the first hallucination score is further based on whether the composite frequency score exceeds the second threshold, and wherein hallucination is likely when the composite frequency score exceeds the second threshold.

3 . The system of claim 1 , wherein the operations further comprise:

parse the first output into a plurality of sections, each section comprising a portion of the first output;

for a first section of the plurality of sections, generate a veracity request indicative of a request to determine a veracity of the first section; and

provide the veracity request of the first section and the target content as third input to the language model.

4 . The system of claim 3 , wherein the operations further comprise:

in response to the providing of the veracity request, receive a third output of the language model, the third output comprising a reply to the veracity request and a corresponding confidence score that the reply indicates veracity; and

based at least in part on the receiving of the third output, generate a third hallucination score that indicates whether hallucination is likely in the first output.

5 . The system of claim 1 , wherein the operations further comprising: determine a set of tokens, comprising at least one token, occurring in the first output that are not found in the target content but are found to occur in the example output; and based at least in part on the determining the set of tokens, for each token of the set of tokens, determine a frequency score indicating occurrence frequency for the token, and wherein the determining of the frequency score comprises dividing the token's occurrence frequency in the example output by a frequency of the token occurring in multiple data sources, the dividing resulting in a quotient, and taking 1 minus the quotient.

6 . The system of claim 1 , wherein the target content includes one of: a meeting transcript, a file, or a document.

7 . The system of claim 1 , wherein the example output corresponds to one-shot or few-shot prompting.

8 . A computer-implemented method comprising:

receiving a first output of a language model based on the language model processing a first input, wherein the first input comprises: a first request, target content, and one or more examples, the one or more examples include one or more model outputs associated with the first request, and wherein the first output comprises a first output confidence and first generative text;

providing a second input to the language model, wherein the second input comprises: the target content and the first output, and wherein the second input excludes any examples;

in response to the providing the second input, receiving a second output from the language model, wherein the second output comprises a second output confidence;

comparing the first output confidence and the second output confidence;

based at least in part on the comparing, generating a first hallucination score that indicates a likelihood that hallucination exists in the first output; and

based at least in part on the first hallucination score exceeding a threshold: refraining from presenting, at a user device, the first generative text, generating second generative text associated with a second hallucination score that does not exceed the threshold, and causing presentation, at the user device, of the second generative text.

9 . The computer-implemented method of claim 8 , further comprising:

determining a set of tokens occurring in the first output that are not found in the target content but are found to occur in at least one example, of the one or more examples;

based at least in part on the determining, for each token of the set of tokens, determining a frequency score indicating how common the token is; and

based at least in part on the determining of the frequency score for each token of the set of tokens, determining a third hallucination score indicating another likelihood that hallucination exists in the first output.

10 . The computer-implemented method of claim 9 , further comprising:

determining a composite frequency score from the frequency score of the set of tokens; and

determining whether the composite frequency score exceeds a second threshold, wherein the generating of the third hallucination score is further based on whether the composite frequency score exceeds the second threshold.

11 . The computer-implemented method of claim 8 , further comprising:

parsing the first output into a plurality of sections, each section comprising a portion of the first output;

for each section of the plurality of sections, generating a veracity request indicative of a request to determine a veracity of the section; and

providing each veracity request for each section and the target content as third input to the language model.

12 . The computer-implemented method of claim 11 , further comprising:

in response to the providing of each veracity request, receiving a third output of the language model, the third output comprises a reply to the veracity request and a corresponding confidence score that the reply is yes; and

based at least in part on the receiving of the third output, generating a third hallucination score that indicates whether hallucination is likely in the first output.

13 . The computer-implemented method of claim 8 , wherein the generating of the first hallucination score comprises:

computing a difference between the first output confidence and the second output confidence;

determining whether the difference exceeds a second threshold; and

generating the first hallucination score based on whether the difference exceeds the second threshold, wherein the hallucination likely exists if the first hallucination score exceeds the second threshold and the hallucination likely does not exist if the first hallucination score does not exceed the second threshold.

14 . One or more computer storage media having computer-executable instructions embodied thereon that, when executed, by one or more processors, cause the one or more processors to perform operations comprising:

receiving a first output of a language model based on the language model processing a first input, wherein the first input comprises a request, target content, and one or more examples, the one or more examples include one or more model outputs associated with the request, and wherein the first output comprises a first output confidence and first generative text;

providing a second input to the language model, wherein the second input comprises: the target content and the first output, and wherein the second input excludes any examples;

in response to the providing the second input, receiving a second output from the language model, wherein the second output comprises a second output confidence;

comparing the first output confidence and the second output confidence;

based at least in part on the comparing, generating a first hallucination score that indicates whether hallucination is likely in the first output; and

based at least in part on the first hallucination score exceeding a threshold, refraining from presenting, at a user device, the first generative text and causing presentation, at the user device, of second generative text associated with a second hallucination score that does not exceed the threshold.

15 . The one or more computer storage media of claim 14 , wherein the generating of the first hallucination score comprises:

determining a composite confidence score by summing, for each section, a corresponding confidence score that a reply confirms veracity;

evaluating a composite frequency score against a second threshold; and

based on the composite frequency score exceeding the second threshold, generating the first hallucination score indicating that there is likely hallucination in the first output.