Evaluation and optimization of natural language to database query translation

1 . A system comprising:

a processor; and

a computer-readable medium storing programming instructions for execution by the processor, the programming instructions, upon execution by the processor, causing the system to perform the following operations:

receiving, by a query translator, a natural language query;

translating, via a prompt generator of the query translator, the natural language query into a first database query based on a database query language;

processing the first database query using a large language model (LLM) to generate a first database query result;

processing a second database query using the LLM to generate a second database query result, wherein the second database query is a natural language equivalent of the first database query that has a different form, expression, or syntax than the first database query;

transforming the first database query result into a first canonical query result and the second database query result into a second canonical query result; and

providing a training adjustment to the query translator in response to the first canonical query result mismatching the second canonical query result, the training adjustment prompting the query translator to adjust the prompt generator in response to the training adjustment.

2 . The system of claim 1 , wherein transforming the first database query result into the first canonical query result comprises:

mapping each row of the first database query result to a column permutation that sorts values in the corresponding row of the first database query result in lexicographic order;

identifying the row of the first database query result that yields a smallest lexicographic order when applying its corresponding permutation;

sorting columns of the first database query result using the column permutation of the identified row of the first database query result; and

sorting rows of the first database query result in lexicographic order to obtain the first canonical query result.

3 . The system of claim 1 , wherein the programming instructions further cause the system to perform the following operations:

constructing a bipartite graph connecting columns of the first database query result with columns of the second database query result; and

identifying a maximum matching in the bipartite graph; and

determining whether the maximum matching covers all columns in the first database query and the second database query result.

4 . The system of claim 1 , wherein the programming instructions further cause the system to perform the following operations:

generating a plurality of databases having differing content;

generating a third database query result with the first database query and a second database of the plurality of databases;

generating a fourth database query result with the second database query and the second database;

determining whether the third database query result matches the fourth database query result;

in response to at least the third database query result matching the fourth database query result, providing training reinforcement for the query translator; and

in response to at least the third database query result not matching the fourth database query result, providing a training adjustment to the query translator.

5 . A method comprising:

receiving, by a query translator, a natural language query;

translating, via a prompt generator of the query translator, the natural language query into a first database query based on a database query language;

processing the first database query using a large language model (LLM) to generate a first database query result;

processing a second database query using the LLM to generate a second database query result, wherein the second database query is a natural language equivalent of the first database query that has a different form, expression, or syntax than the first database query;

transforming the first database query result into a first canonical query result and the second database query result into a second canonical query result; and

providing a training adjustment to the query translator in response to the first canonical query result mismatching the second canonical query result, the training adjustment prompting the query translator to adjust the prompt generator in response to the training adjustment.

6 . The method of claim 1 , wherein transforming the first database query result into the first canonical query result comprises:

mapping each row of the first database query result to a column permutation that sorts values in the corresponding row of the first database query result in lexicographic order;

identifying the row of the first database query result that yields a smallest lexicographic order when applying its corresponding permutation;

sorting columns of the first database query result using the column permutation of the identified row of the first database query result; and

sorting rows of the first database query result in lexicographic order to obtain the first canonical query result.

7 . The method of claim 5 , further comprising

constructing a bipartite graph connecting columns of the first database query result with columns of the second database query result; and

identifying a maximum matching in the bipartite graph; and

determining whether the maximum matching covers all columns in the first database query and the second database query result.

8 . The method of claim 5 , further comprising:

generating a plurality of databases having differing content;

generating a third database query result with the first database query and a second database of the plurality of databases;

generating a fourth database query result with the second database query and the second database;

determining whether the third database query result matches the fourth database query result;

in response to at least the third database query result matching the fourth database query result, providing training reinforcement for the query translator; and

in response to at least the third database query result not matching the fourth database query result, providing a training adjustment to the query translator.

9 . A computer storage device storing programming instructions for execution by a processor of a system, the programming instructions, upon execution by the processor, causing the system to perform the following operations:

receiving, by a query translator, a natural language query;

translating, via a prompt generator of the query translator, the natural language query into a first database query based on a database query language;

processing the first database query using a large language model (LLM) to generate a first database query result;

processing a second database query using the LLM to generate a second database query result, wherein the second database query is a natural language equivalent of the first database query that has a different form, expression, or syntax than the first database query;

transforming the first database query result into a first canonical query result and the second database query result into a second canonical query result; and

providing a training adjustment to the query translator in response to the first canonical query result mismatching the second canonical query result, the training adjustment prompting the query translator to adjust the prompt generator in response to the training adjustment.

10 . The computer storage device of claim 9 , wherein the programming instructions further cause the system to perform the following operations

constructing a bipartite graph connecting columns of the first database query result with columns of the second database query result; and

identifying a maximum matching in the bipartite graph, wherein determining whether the first database query result matches the second database query result comprises:

determining whether the maximum matching covers all columns in the first database query and the second database query result.

11 . The computer storage device of claim 9 , wherein the programming instructions further cause the system to perform the following operations:

generating a plurality of databases having differing content;

generating a third database query result with the first database query and a second database of the plurality of databases;

generating a fourth database query result with the second database query and the second database;

determining whether the third database query result matches the fourth database query result;

in response to at least the third database query result matching the fourth database query result, providing training reinforcement for the query translator; and

in response to at least the third database query result not matching the fourth database query result, providing a training adjustment to the query translator.

12 . The system of claim 1 , wherein the second database query is a known good database query.

13 . The system of claim 1 , wherein the second database query is a curated database query.

14 . The system of claim 1 , wherein the training adjustment prompts the query translator to reverse a prior training adjustment of the prompt generator.

15 . The system of claim 1 , wherein the programming instructions further cause the system to perform the following operation:

translating, via the adjusted prompt generator of the query translator, a subsequent natural language query into a subsequent database query based on the database query language.

16 . The system of claim 1 , where the query translator includes a machine learning (ML) language model with a transformer architecture.

17 . The system of claim 16 , where the query translator includes a generative pre-trained transformer.