Text-to-structured query language query generation using query logs and data profiling
In one example, a method performed by a processing system including at least one processor includes automatically inferring metadata about a structured query language database, constructing, based on the metadata, a prompt for generating a new structured query language query to be executed against the structured query language database, and providing the prompt to a language model for use in generating the new structured query language query.
1 . A method comprising:
performing, by a processing system including at least one processor, a statistical analysis over a plurality of fields of a plurality of tables contained in a structured query language database to generate a profile for the structured query language database, wherein the statistical analysis comprises:
collecting, by the processing system, a plurality of sample values from a field of the plurality of fields, wherein the plurality of sample values is collected using an approximation technique to minimize an amount of memory needed to store the plurality of sample values;
submitting, by the processing system to a language model, a description of contents of a table of the plurality of tables in which the field is contained, a name of the field, and a request to summarize the plurality of sample values; and
receiving, by the processing system from the language model, a summarization of the field, wherein the summarization of the field is used to generate a prompt; and
constructing, by the processing system, the prompt for the language model, wherein the language model is trained to generate a new structured query language query to be executed against the structured query language database, using data from the profile.
2 . The method of claim 1 , wherein the performing statistical analysis comprises determining:
a number of records in a table of the plurality of tables.
3 . The method of claim 1 , wherein the plurality of values comprises at least one of: a count of distinct values contained in the each field or a most frequent value contained in the each field.
4 . The method of claim 1 , wherein the performing statistical analysis comprises determining, for each field of the plurality of fields that is contained in the table, a count of non-null values contained in the each field.
5 . The method of claim 1 , wherein the performing statistical analysis comprises determining, for each field of the plurality of fields that is contained in the table, a count of distinct values contained in the each field.
6 . The method of claim 1 , wherein the performing statistical analysis comprises determining, for each field of the plurality of fields that is contained in the table, a minimum value contained in the each field.
7 . The method of claim 1 , wherein the performing statistical analysis comprises determining, for each field of the plurality of fields that is contained in the table, a maximum value contained in the each field.
8 . The method of claim 1 , wherein the performing statistical analysis comprises determining, for each field of the plurality of fields that is contained in the table, a most frequent value contained in the each field.
9 . The method of claim 8 , wherein a skewness of the each field is estimated from the most frequent value contained in the each field.
10 . The method of claim 9 , wherein the each field is assumed to be skewed when a top number of most frequent values occur in the each field at least a threshold number of times that is expected from a uniform distribution of values.
11 . The method of claim 1 , wherein the performing statistical analysis comprises determining a data pattern occurring across the plurality of fields.
12 . The method of claim 11 , wherein the data pattern comprises an observed typical field length.
13 . The method of claim 11 , wherein the data pattern comprises a use of uppercase versus lowercase characters.
14 . The method of claim 11 , wherein the data pattern comprises an occurrence of characters that are neither alphabetic nor numeric.
15 . The method of claim 11 , wherein the data pattern comprises repeated prefixes or suffixes.
16 . The method of claim 1 , wherein the performing statistical analysis comprises determining, for each pair of fields in the plurality of fields, a resemblance between the each pair of fields.
17 . The method of claim 16 , wherein the resemblance is calculated as a min hash.
18 . The method of claim 17 , wherein the min hash is expanded to estimate a containment of the each pair of fields.
19 . A non-transitory computer readable medium storing instructions which, when executed by a processing system including at least one processor, cause the processing system to perform operations, the operations comprising:
performing a statistical analysis over a plurality of fields of a plurality of tables contained in a structured query language database to generate a profile for the structured query language database, wherein the statistical analysis comprises:
collecting a plurality of sample values from a field of the plurality of fields, wherein the plurality of sample values is collected using an approximation technique to minimize an amount of memory needed to store the plurality of sample values;
submitting, to a language model, a description of contents of a table of the plurality of tables in which the field is contained, a name of the field, and a request to summarize the plurality of sample values; and
receiving, from the language model, a summarization of the field, wherein the summarization of the field is used to generate a prompt; and
constructing the prompt for the language model, wherein the language model is trained to generate a new structured query language query to be executed against the structured query language database, using data from the profile.
20 . An apparatus comprising:
a processing system including at least one processor; and
a non-transitory computer readable medium storing instructions which, when executed by the processing system, cause the processing system to perform operations, the operations comprising:
performing a statistical analysis over a plurality of fields of a plurality of tables contained in a structured query language database to generate a profile for the structured query language database, wherein the statistical analysis comprises:
collecting a plurality of sample values from a field of the plurality of fields, wherein the plurality of sample values is collected using an approximation technique to minimize an amount of memory needed to store the plurality of sample values;
submitting, to a language model, a description of contents of a table of the plurality of tables in which the field is contained, a name of the field, and a request to summarize the plurality of sample values; and
receiving, from the language model, a summarization of the field, wherein the summarization of the field is used to generate a prompt; and
constructing the prompt for the language model, wherein the language model is trained to generate a new structured query language query to be executed against the structured query language database, using data from the profile.