Granted Patent B2

US 12,468,898 · App. 18/144,802 · Granted Nov 11, 2025

Inventors: Shima Imani (Sammamish, WA); Harsh Shrivastava (Redmond, WA); Liang Du (Redmond, WA)

Assignee: MICROSOFT TECHNOLOGY LICENSING, LLC

G06F40/40G06F16/3325G06F16/3329

Abstract

Claims (105)

1 . A method for an artificial intelligence (AI) system with a large language model (LLM) to solve a mathematical problem, the method comprising:

receiving an initial query that presents a problem with original input values;

creating key-value mappings between the original input values and variables;

transforming the initial query into a template query by replacing the original input values with the variables;

sending multiple prompts to the LLM, wherein each of the multiple prompts is different and contextually related to the template query;

responsive to the multiple prompts, receiving multiple results from the LLM, wherein each of the multiple results includes an analytical expression to solve the mathematical problem;

evaluating outputs of the analytical expressions included in the multiple results with the variables being assigned to a common set of randomly sampled values, wherein evaluating the outputs of the analytical expressions included in the multiple results comprises:

looping through a process over a number of trials, the process comprising:

assigning random values to the variables;

evaluating each of the analytical expressions with the variables having the random values assigned;

calculating a consensus rating based on the evaluating each of the analytical expressions with the variables having the random values assigned;

determining if additional trials are required based on the consensus rating and a test condition; and

terminating the looping when the consensus rating and the test condition indicate that the additional trials are not required; and

outputting final results based on the consensus rating and the test condition indicating that the additional trials are not required.

2 . The method of claim 1 , further comprising relaxing the test condition for terminating the loop when a non-convergence condition is detected.

3 . The method of claim 1 , wherein the test condition comprises one of:

terminating the loop when the number of trials is reached;

terminating the loop when all of the evaluated outputs are matched;

terminating the loop when all of the evaluated outputs are matched over a specified percentage;

terminating the loop when all of the evaluated outputs are matched over a specified number of consecutive trials; or

terminating the loop when all of the evaluated outputs are matched over the specified percentage for the specified number of consecutive trials.

4 . The method of claim 1 , wherein outputting the final results comprises:

assigning the original input values to the variables;

selecting one or more of the analytical expressions for evaluation;

evaluating outputs of each selected analytical expression with the variables having the original input values assigned; and

providing the evaluated outputs of each selected analytical expression.

5 . The method of claim 4 , wherein selecting one or more of the analytical expressions for evaluation comprises one of:

selecting a single analytical expression;

selecting all analytical expressions;

selecting a single analytical expression above a specified confidence rating;

selecting all analytical expressions above a specified confidence rating;

selecting a first analytical expression from all analytical expressions in a consensus from a most recent iteration;

randomly picking a single analytical expression from all analytical expressions in the consensus from the most recent iteration;

ranking all the analytical expressions in the consensus from the most recent iteration, and selecting a highest ranked analytical expression; and

ranking all the analytical expressions in the consensus from the most recent iteration, and selecting all ranked analytical expressions above a specified rating.

6 . A computer-readable storage medium having computer-executable instructions stored thereupon that, when executed by one or more processing units of an artificial intelligence (AI) system with a large language model (LLM), cause the AI system to:

receive an initial query that presents a problem with original input values;

transform the initial query into a template query by replacing the original input values with variables;

send multiple prompts to the LLM, wherein each of the multiple prompts is different and contextually related to the template query;

responsive to the multiple prompts, receive multiple results from the LLM, wherein each of the multiple results includes an analytical expression to solve the mathematical problem;

verify results from an evaluation of outputs of the analytical expressions included in the multiple results with the variables being assigned to a common set of randomly sampled values, wherein verifying the results from the evaluation of the outputs of the analytical expressions included in the multiple results comprises:

looping through a process over a number of trials, the process comprising:

assigning random values to the variables;

evaluating each of the analytical expressions with the variables having the random values assigned;

calculating a consensus rating based on the evaluating each of the analytical expressions with the variables having the random values assigned;

determining if additional trials are required based on the consensus rating and a test condition; and

terminating the looping when the consensus rating and the test condition indicate that the additional trials are not required; and

output final results based on the consensus rating and the test condition indicating that the additional trials are not required.

7 . The computer-readable storage medium of claim 6 , wherein the computer-executable instructions further cause the AI system to relax the test condition for terminating the loop when a non-convergence condition is detected.

8 . The computer-readable storage medium of claim 6 , wherein the test condition comprises one of:

terminating the loop when the number of trials is reached;

terminating the loop when all of the evaluated outputs are matched;

terminating the loop when all of the evaluated outputs are matched over a specified percentage;

terminating the loop when all of the evaluated outputs are matched over a specified number of consecutive trials; or

terminating the loop when all of the evaluated outputs are matched over the specified percentage for the specified number of consecutive trials.

9 . The computer-readable storage medium of claim 6 , wherein outputting the final results comprises:

assigning the original input values to the variables;

selecting one or more of the analytical expressions for evaluation;

evaluating outputs of each selected analytical expression with the variables having the original input values assigned; and

providing the evaluated outputs of each of the selected analytical expression.

10 . The computer-readable storage medium of claim 9 , wherein selecting one or more of the analytical expressions for evaluation comprises one of:

selecting a single analytical expression;

selecting all analytical expressions;

selecting a single analytical expression above a specified confidence rating;

selecting all analytical expressions above a specified confidence rating;

selecting a first analytical expression from all analytical expressions in a consensus from a most recent iteration;

randomly picking a single analytical expression from all analytical expressions in the consensus from the most recent iteration;

ranking all the analytical expressions in the consensus from the most recent iteration, and selecting a highest ranked analytical expression; and

ranking all the analytical expressions in the consensus from the most recent iteration, and selecting all ranked analytical expressions above a specified rating.

11 . An artificial intelligence (AI) system, comprising:

a processor; and

a computer-readable storage medium having computer-executable instructions stored thereupon that, when executed by the processor, cause the AI system to:

receive an initial query that presents a problem with original input values;

transform the initial query into a template query by replacing the original input values with variables;

send multiple prompts to the LLM, wherein each of the multiple prompts is and contextually related to the template query;

responsive to the multiple prompts, receive multiple results from the LLM, wherein each of the multiple results includes an analytical expression to solve the mathematical problem;

verify results from an evaluation of outputs of the analytical expressions included in the multiple results with the variables being assigned to a common set of randomly sampled values, wherein verifying the results from the evaluation of the outputs of the analytical expressions included in the multiple results comprises:

looping through a process over a number of trials, the process comprising:

assigning random values to the variables;

evaluating each of the analytical expressions with the variables having the random values assigned;

calculating a consensus rating based on the evaluating each of the analytical expressions with the variables having the random values assigned;

determining if additional trials are required based on the consensus rating and a test condition; and

terminating the looping when the consensus rating and the test condition indicate that the additional trials are not required; and

output final results based on the consensus rating and the test condition indicating that the additional trials are not required.

12 . The AI system of claim 11 , wherein the computer-executable instructions further cause the AI system to relax the test condition for terminating the loop when a non-convergence condition is detected.

13 . The AI system of claim 11 , wherein the test condition comprises one of:

terminating the loop when the number of trials is reached;

terminating the loop when all of the evaluated outputs are matched;

terminating the loop when all of the evaluated outputs are matched over a specified percentage;

terminating the loop when all of the evaluated outputs are matched over a specified number of consecutive trials; or

terminating the loop when all of the evaluated outputs are matched over the specified percentage for the specified number of consecutive trials.

14 . The AI system of claim 11 , wherein outputting the final results comprises:

assigning the original input values to the variables;

selecting one or more of the analytical expressions for evaluation;

evaluating outputs of each selected analytical expression with the variables having the original input values assigned; and

providing the evaluated outputs of each of the selected analytical expression.

15 . The AI system of claim 14 , wherein selecting one or more of the analytical expressions for evaluation comprises one of:

selecting a single analytical expression;

selecting all analytical expressions;

selecting a single analytical expression above a specified confidence rating;

selecting all analytical expressions above a specified confidence rating;

selecting a first analytical expression from all analytical expressions in a consensus from a most recent iteration;

randomly picking a single analytical expression from all analytical expressions in the consensus from the most recent iteration;

ranking all the analytical expressions in the consensus from the most recent iteration, and selecting a highest ranked analytical expression; and

ranking all the analytical expressions in the consensus from the most recent iteration, and selecting all ranked analytical expressions above a specified rating.

Assignments (1)

Continuity (2)

References Cited (33)