Numerically accurate generative machine learning system

1 . A system, comprising:

one or more computing devices configured to implement a generative machine learning system, configured to:

receive a question comprising at least a portion of which requires mathematical operations to be performed using data from one or more data sources, wherein the one or more data sources comprise numerical data;

generate, using a generative machine learning model, instructions for a deterministic computational engine to perform a deterministic query for answering the at least a portion of the question that requires mathematical operations using the data from the one or more data sources, wherein the deterministic query produces a same output when given a same input using the deterministic computational engine;

verify that instructions for the deterministic computational engine to perform the deterministic query are correct based on an expected structure for the mathematical operations used in the deterministic query;

perform, using the deterministic computational engine, the deterministic query using the data from the one or more data sources;

generate the answer for the question based on the question and the result of the deterministic query, wherein at least a portion of the answer is associated with the at least a portion of the question that requires mathematical operations;

verify that the at least a portion of the answer associated with the at least a portion of the question that requires mathematical operations is numerically consistent with corresponding portions of the result of the deterministic query received from the deterministic computational engine; and

provide, in response to the question, the answer that has been verified.

2 . The system of claim 1 , wherein the one or more computing devices configured to implement the generative machine learning system is configured to:

determine whether the result of the deterministic query meets a threshold criteria for implementing a fenced region in an input to the generative machine learning model, wherein the fenced region in the input may not be altered by the generative machine learning model;

based on the determination that the result of the deterministic query meets the threshold criteria, modify the result of the deterministic query provided to the generative machine learning model with an opening token and a closing token denoting the fenced region encapsulating at least a portion of the result of the deterministic query; and

wherein the result of the deterministic query provided to the generative machine learning model is the modified result of the deterministic query comprising the opening token and the closing token denoting the fenced region.

3 . The system of claim 1 , wherein the one or more computing devices configured to implement the generative machine learning system is configured to:

determine, using the generative machine learning model, that a given portion of the data from the one or more data sources are relevant to the performance of the question regarding the one or more data sources;

generate, using the generative machine learning model, one or more applicational programming interface (API) calls to retrieve the given portion of the data from the one or more data sources for performing the deterministic query; and

retrieve, using the generated one or more API calls, the given portion of the data from the one or more data sources; and

format, using the generative machine learning model, the retrieved given portion of the data according to a schema associated with the deterministic computational engine.

4 . The system of claim 1 , wherein the one or more computing devices configured to implement the generative machine learning system is configured to:

determine a given domain type associated with the question; and

wherein the generative machine learning system is configured to verify that the instructions for the deterministic computational engine to perform the deterministic query are correct using a library of domain-specific deterministic queries of the given domain type, wherein the library comprises structures for mathematical operations for the given domain type.

5 . The system of claim 1 , wherein the one or more computing devices configured to implement the generative machine learning system is configured to:

provide, to the generative machine learning model, a failed result of the deterministic query to the generative machine learning model; and

generate, based on the failed result of the deterministic query, another instruction for the deterministic computational engine to perform another deterministic query, wherein the other deterministic query corrects for deficiencies of the deterministic query contributing to the failed result.

6 . The system of claim 1 , wherein:

the deterministic computational engine is a structured query language (SQL)-based computational engine; and

the deterministic query is a SQL query comprising SQL operations that are deterministically verifiable.

7 . A method, comprising:

receiving a question comprising at least a portion of which requires mathematical operations to be performed using data from one or more data sources, wherein the one or more data sources comprise numerical data;

generating, using a generative machine learning model, instructions for a deterministic computational engine to perform a deterministic query for answering the at least a portion of the question that requires mathematical operations using the data from the one or more data sources, wherein the deterministic query produces a same output when given a same input using the deterministic computational engine;

verifying that instructions for the deterministic computational engine to perform the deterministic query are correct based on an expected structure for the mathematical operations used in the deterministic query;

performing, using the deterministic computational engine, the deterministic query using the data from the one or more data sources;

generating the answer for the question based on the question and the result of the deterministic query, wherein at least a portion of the answer is associated with the at least a portion of the query that requires mathematical operations;

verifying that the at least a portion of the answer associated with the at least a portion of the question that requires mathematical operations is numerically consistent with corresponding portions of the result of the deterministic query received from the deterministic computational engine; and

providing, in response to the question, the answer that has been verified.

8 . The method of claim 7 , further comprising:

determining whether the result of the deterministic query meets a threshold criteria for implementing a fenced region in an input to the generative machine learning model, wherein the fenced region in the input may not be altered by the generative machine learning model;

based on the determination that the result of the deterministic query meets the threshold criteria, modifying the result of the deterministic query provided to the generative machine learning model with an opening token and a closing token denoting the fenced region encapsulating at least a portion of the result of the deterministic query; and

wherein the result of the deterministic query provided to the generative machine learning model is the modified result of the deterministic query comprising the opening token and the closing token denoting the fenced region.

9 . The method of claim 7 , further comprising:

determining, using the generative machine learning model, that a given portion of the data from the one or more data sources are relevant to the performance of the question regarding the one or more data sources;

generating, using the generative machine learning model, one or more applicational programming interface (API) calls to retrieve the given portion of the data from the one or more data sources for performing the deterministic query; and

retrieving, using the generated one or more API calls, the given portion of the data from the one or more data sources; and

formatting, using the generative machine learning model, the retrieved given portion of the data according to a schema associated with the deterministic computational engine.

10 . The method of claim 9 , further comprising:

verifying, using the generative machine learning model, that the one or more API calls to retrieve the given portion of the data are correct using API documentation for the one or more data sources.

11 . The method of claim 7 , further comprising:

determining a given domain type associated with the question; and

verifying that the instructions for the deterministic computational engine to perform the deterministic query are correct using a library of domain-specific deterministic queries of the given domain type, wherein the library comprises structures for mathematical operations for the given domain type.

12 . The method of claim 7 , further comprising:

providing, to the generative machine learning model, a failed result of the deterministic query to the generative machine learning model; and

generating, based on the failed result of the deterministic query, another instruction for the deterministic computational engine to perform another deterministic query, wherein the other deterministic query corrects for deficiencies of the deterministic query contributing to the failed result.

13 . The method of claim 7 , wherein:

the deterministic computational engine is a structured query language (SQL)-based computational engine; and

the deterministic query is a SQL query comprising SQL operations that are deterministically verifiable.

14 . One or more non-transitory, computer-readable storage media, storing program instructions that when executed on or across one or more computing devices cause the one or more computing devices to implement:

receiving a question comprising at least a portion of which requires mathematical operations to be performed using data from one or more data sources, wherein the one or more data sources comprise numerical data;

generating, using a generative machine learning model, instructions for a deterministic computational engine to perform a deterministic query for answering the at least a portion of the question that requires mathematical operations using the data from the one or more data sources, wherein the deterministic query produces a same output when given a same input using the deterministic computational engine;

verifying that instructions for the deterministic computational engine to perform the deterministic query are correct based on an expected structure for the mathematical operations used in the deterministic query;

performing, using the deterministic computational engine, the deterministic query using the data from the one or more data sources;

generating the answer for the question based on the question and the result of the deterministic query, wherein at least a portion of the answer is associated with the at least a portion of the question that requires mathematical operations;

verifying that the at least a portion of the answer associated with the at least a portion of the question that requires mathematical operations is numerically consistent with corresponding portions of the result of the deterministic query received from the deterministic computational engine; and

providing, in response to the question, the answer that has been verified.

15 . The one or more non-transitory, computer-readable, storage media of claim 14 , wherein the program instructions, when executed on or across one or more computing devices, cause the one or more computing devices to implement:

determining whether the result of the deterministic query meets a threshold criteria for implementing a fenced region in an input to the generative machine learning model, wherein the fenced region in the input may not be altered by the generative machine learning model;

based on the determination that the result of the deterministic query meets the threshold criteria, modifying the result of the deterministic query provided to the generative machine learning model with an opening token and a closing token denoting the fenced region encapsulating at least a portion of the result of the deterministic query; and

wherein the result of the deterministic query provided to the generative machine learning model is the modified result of the deterministic query comprising the opening token and the closing token denoting the fenced region.

16 . The one or more non-transitory, computer-readable, storage media of claim 14 , wherein the program instructions, when executed on or across one or more computing devices, cause the one or more computing devices to implement:

determining, using the generative machine learning model, that a given portion of the data from the one or more data sources are relevant to the performance of the question regarding the one or more data sources;

generating, using the generative machine learning model, one or more applicational programming interface (API) calls to retrieve the given portion of the data from the one or more data sources for performing the deterministic query; and

retrieving, using the generated one or more API calls, the given portion of the data from the one or more data sources; and

formatting, using the generative machine learning model, the retrieved given portion of the data according to a schema associated with the deterministic computational engine.

17 . The one or more non-transitory, computer-readable, storage media of claim 16 , wherein the program instructions, when executed on or across one or more computing devices, cause the one or more computing devices to implement:

verifying, using the generative machine learning model, that the one or more API calls to retrieve the given portion of the data are correct using API documentation for the one or more data sources.

18 . The one or more non-transitory, computer-readable, storage media of claim 14 , wherein the program instructions, when executed on or across one or more computing devices, cause the one or more computing devices to implement:

determining a given domain type associated with the question; and

verifying that the instructions for the deterministic computational engine to perform the deterministic query are correct using a library of domain-specific deterministic queries of the given domain type, wherein the library comprises structures for mathematical operations for the given domain type.

19 . The one or more non-transitory, computer-readable, storage media of claim 14 , wherein the program instructions, when executed on or across one or more computing devices, cause the one or more computing devices to implement:

providing, to the generative machine learning model, a failed result of the deterministic query to the generative machine learning model; and

generating, based on the failed result of the deterministic query, another instruction for the deterministic computational engine to perform another deterministic query, wherein the other deterministic query corrects for deficiencies of the deterministic query contributing to the failed result.

20 . The one or more non-transitory, computer-readable, storage media, storing the program instructions of claim 14 , wherein:

the deterministic computational engine is a structured query language (SQL)-based computational engine; and

the deterministic query is a SQL query comprising SQL operations that are deterministically verifiable.