Granted Patent B2

US 12,639,355 · App. 18/238,040 · Granted May 26, 2026

Inventors: William Blum (Bellevue, WA); Amir Hossein Abdi (Gatineau, CA); Martin Fontaine (Vancouver, CA)

Assignee: Microsoft Technology Licensing, LLC

G06F16/345G06F16/3344G06F16/338

Abstract

Claims (44)

1 . A computer-implemented method of verifying a query result provided by a large language model, LLM, in response to a query including input data from which the query result was derived, the method comprising executing, by a computer processor, stored processor-executable instructions to perform operation comprising:

automatically injecting, into a prompt including query text and the input data, a verification request for the query result, wherein the verification request comprises instructions which, when processed by the LLM, cause the LLM to generate verification data that indicates: a value included in the query result, a corresponding source indicating a part of the input data from which the value was derived, and a corresponding evaluable expression comprising a code snippet in a computer-readable expression language for deriving the value from the source;

providing the prompt as input to the LLM;

receiving the query result and verification data generated by the LLM in response to the prompt; and

processing the verification data to determine whether the query result was validly derived from the input data by:

instantiating an evaluation context by loading contextual data into memory, the contextual data comprising the indicated part of the input data;

executing or parsing the code snippet, based on the indicated part of the input data, in the computer-readable expression language to provide an expression output;

comparing the expression output to the value included in the query result; and

in response to the expression output matching the value included in the query result, determining that the value was validly derived from the input data.

2 . The method of claim 1 , wherein automatically injecting the verification requested into the prompt comprises:

automatically selecting a stored verification request text from a plurality of stored verification request texts, based on the query.

3 . The method of claim 1 , wherein the input data is structured data.

4 . The method of claim 1 , wherein the prompt is a second prompt, the method further comprising, prior to generating the second prompt:

generating a first prompt for the LLM, the first prompt including the query; and

providing the first prompt as input to the LLM to cause the LLM to generate the query result.

5 . The method of claim 1 , further comprising extracting the input data from a data store associated with a security system.

6 . The method of claim 1 , wherein the query text comprises instructions which, when processed by the LLM, cause the LLM to summarize the input data.

7 . The method of claim 1 , comprising:

executing or parsing the code snippet in an isolated computational environment.

8 . The method of claim 1 , comprising:

displaying, on a user interface, the query result; and

highlighting the value included in the query result according to an output of the comparing the expression output to the value.

9 . The method of claim 1 , comprising:

in response to the expression output not matching the value included in the query result, generating a tagged query result including a tag indicating that the value is hallucinated;

generating a prompt including the tagged query result and instructions that, when processed by the LLM, cause the LLM to rewrite the query result with the value omitted;

inputting the prompt to the LLM; and

receiving in response a rewritten query result omitting the value.

10 . The method of claim 1 , comprising:

in response to the expression output not matching the value included in the query result, generating a prompt including:

the query result;

the evaluable expression;

the value included in the query result;

an indication that the value could not be derived from the input data, and

instructions that, when processed by the LLM, cause the LLM to generate a corrected query result.

11 . A computer system for verifying a query result provided by a large language model, LLM, in response to a query including input data from which the query result was derived, the computer system comprising a processor and a memory, the memory storing instructions, which when executed by the processor, cause the system to:

automatically inject, into a prompt including query text and the input data, a verification request for the query result, wherein the verification request comprises instructions which, when processed by the LLM, cause the LLM to generate verification data that indicates: a value included in the query result, a corresponding source indicating a part of the input data from which the value was derived, and a corresponding evaluable expression comprising a code snippet in a computer-readable expression language for deriving the value from the source; and

provide the prompt as input to the LLM;

receive the query result and verification data generated by the LLM in response to the prompt;

process the verification data to determine whether the query result was validly derived from the input data by:

instantiating an evaluation context by loading contextual data into memory, the contextual data comprising the indicated part of the input data;

executing or parsing the code snippet, based on the indicated part of the input data, in the computer-readable expression language to provide an expression output;

comparing the expression output to the value included in the query result; and

in response to the expression output matching the value included in the query result determine that the value was validly derived from the input data.

12 . The computer system of claim 11 , wherein the input data is based on data extracted from a data store associated with a security system, and the query text comprises instructions which, when processed by the LLM, cause the LLM to summarize the input data.

Assignments (1)

Continuity (1)

References Cited (11)