Large language model and deterministic calculator systems and methods
A first large language model (LLM) instance may be instructed to request data while being prevented from performing calculations using the data. A second LLM instance may be instructed to provide a response to the request for data based on a known complete data set. The response may be translated into a machine-readable response in a format configured for processing by a calculation engine. The calculation engine may process the machine-readable response, thereby generating a calculation engine output. A mismatch between the calculation engine output and a known result obtained using the known complete data set may be identified, and the instruction to the first LLM may be modified in response.
1 . A method comprising:
generating, by at least one processor, a first test instruction, the first test instruction configured to:
cause a first large language model (LLM) instance to request data through an LLM dialog session, the data including information to be processed by a calculation engine,
while preventing the first LLM instance from performing calculations that produce a mathematical output using the data;
generating, by the at least one processor, a second test instruction configured to cause a second LLM instance to respond to the request, wherein the second test instruction includes a known complete data set;
sending the first test instruction to the first LLM instance, thereby causing the first LLM instance to request the data from the second LLM instance;
in response to the processing of the first test instruction, receiving a response to the request for data generated by the second LLM instance, the response including at least a portion of the data;
translating the response into a machine-readable response in a format configured for processing by the calculation engine executed by the at least one processor;
processing, by the calculation engine executed by the at least one processor, the machine-readable response, the processing including performing the calculations, thereby generating a calculation engine output that includes the mathematical output;
identifying, by the at least one processor, a mismatch between the calculation engine output and a known result obtained using the known complete data set; and
modifying, by the at least one processor, the first test instruction in response to the mismatch, the modifying including changing at least a portion of information included in the request for the data.
2 . The method of claim 1 , wherein:
the receiving comprises determining that the response includes less than all of the data;
the first LLM instance makes at least one additional request for at least a portion of remaining data and the second LLM instance makes at least one additional response to the at least one additional request; and
the translating includes translating the response and the at least one additional response.
3 . The method of claim 1 , wherein:
the data comprises multiple parts; and
the first test instruction is further configured to cause the first LLM instance to attempt to obtain a plurality of the multiple parts in a single response.
4 . The method of claim 1 , wherein the translating comprises:
generating, by the at least one processor, a translation instruction configured to cause a third LLM instance to convert the response into the machine-readable response; and
receiving, by the at least one processor, the machine-readable response from the third LLM instance.
5 . The method of claim 1 , wherein the translating comprises applying, by the at least one processor, a data extraction model to the response, thereby generating the machine-readable response.
6 . The method of claim 1 , further comprising processing, by the calculation engine executed by the at least one processor, the known complete data set, thereby generating the known result.
7 . A method comprising:
generating, by at least one processor, a first test instruction, the first test instruction configured to:
cause a first large language model (LLM) instance to request data through an LLM dialog session, the data including information to be processed by a calculation engine,
while preventing the first LLM instance from performing calculations that produce a mathematical output using the data;
generating, by the at least one processor, a second test instruction configured to cause a second LLM instance to respond to the request, wherein the second test instruction includes a known complete data set;
processing, by the first LLM instance executed by the at least one processor, the first test instruction to thereby cause the first LLM instance to request the data from the second LLM instance;
in response to the processing of the first test instruction, processing, by the second LLM instance executed by the at least one processor, the second test instruction to thereby provide a response to the request for data generated by the first LLM instance, the response including at least a portion of the data;
translating the response into a machine-readable response in a format configured for processing by the calculation engine executed by the at least one processor;
processing, by the calculation engine executed by the at least one processor, the machine-readable response, the processing including performing the calculations, thereby generating a calculation engine output that includes the mathematical output;
identifying, by the at least one processor, a mismatch between the calculation engine output and a known result obtained using the known complete data set; and
modifying, by the at least one processor, the first test instruction in response to the mismatch, the modifying including changing at least a portion of information included in the request for the data.
8 . The method of claim 7 , wherein:
the receiving comprises determining that the response includes less than all of the data;
the first LLM instance makes at least one additional request for at least a portion of remaining data and the second LLM instance makes at least one additional response to the at least one additional request; and
the translating includes translating the response and the at least one additional response.
9 . The method of claim 7 , wherein:
the data comprises multiple parts; and
the first test instruction is further configured to cause the first LLM instance to attempt to obtain a plurality of the multiple parts in a single response.
10 . The method of claim 7 , wherein the translating comprises:
generating, by the at least one processor, a translation instruction configured to cause a third LLM instance to convert a user-generated response into the machine-readable response; and
processing, by the third LLM instance executed by the at least one processor, the translation instruction to thereby convert the response into the machine-readable response.
11 . The method of claim 7 , further comprising processing, by the calculation engine executed by the at least one processor, the known complete data set, thereby generating the known result.
12 . A system comprising:
at least one processor; and
at least one non-transitory computer-readable memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform processing comprising:
generating, by at least one processor, a first test instruction, the first test instruction configured to:
cause a first large language model (LLM) instance to request data through an LLM dialog session, the data including information to be processed by a calculation engine,
while preventing the first LLM instance from performing calculations that produce a mathematical output using the data;
generating a second test instruction configured to cause a second LLM instance to respond to the request, wherein the second test instruction includes a known complete data set;
sending the first test instruction to the first LLM instance, thereby causing the first LLM instance to request the data from the second LLM instance;
in response to the processing of the first test instruction, receiving a response to the request for data generated by the second LLM instance, the response including at least a portion of the data;
translating the response into a machine-readable response in a format configured for processing by the calculation engine executed by the at least one processor;
processing, by the calculation engine, the machine-readable response, the processing including performing the calculations, thereby generating a calculation engine output that includes the mathematical output;
identifying a mismatch between the calculation engine output and a known result obtained using the known complete data set; and
modifying the first test instruction in response to the mismatch, the modifying including changing at least a portion of information included in the request for the data.
13 . The system of claim 12 , wherein:
the receiving comprises determining that the response includes less than all of the data;
the first LLM instance makes at least one additional request for at least a portion of remaining data and the second LLM instance makes at least one additional response to the at least one additional request; and
the translating includes translating the response and the at least one additional response.
14 . The system of claim 12 , wherein:
the data comprises multiple parts; and
the first test instruction is further configured to cause the first LLM instance to attempt to obtain a plurality of the multiple parts in a single response.
15 . The system of claim 12 , wherein the translating comprises:
generating a translation instruction configured to cause a third LLM instance to convert the response into the machine-readable response; and
receiving the machine-readable response from the third LLM instance.
16 . The system of claim 12 , wherein the translating comprises applying a data extraction model to the response, thereby generating the machine-readable response.
17 . The system of claim 12 , wherein the processing further comprises processing, by the calculation engine, the known complete data set, thereby generating the known result.
18 . The system of claim 12 , wherein the processing further comprises:
processing, by the first LLM instance, the first test instruction to thereby provide the request for data to the second LLM instance;
processing, by the second LLM instance, the second test instruction to thereby provide a response to the request for data generated by the second LLM instance, the response including at least a portion of the data.
19 . The system of claim 12 , wherein the translating comprises:
generating a translation instruction configured to cause a third LLM instance to convert a user-generated response into the machine-readable response; and
processing, by the third LLM instance, the translation instruction to thereby convert the response into the machine-readable response.
20 . The system of claim 12 , wherein the at least one processor comprises a first processor configured to perform the processing and a second processor configured to operate the first LLM instance and the second LLM instance.