In-context learning for NL2SQL with pattern based retrieval
The present disclosure relates to machine learning techniques for In-Context-Learning (ICL) with pattern-based retrieval for the task of converting Natural Language (NL) to Structured Query Language (SQL). Aspects are directed towards acquiring a natural language utterance and a database schema, searching, using at least a portion of the natural language utterance as a key, a memory bank for one or more in-context examples that are relevant to the key, generating a prompt comprising the natural language query, the database schema, and the one or more in-context examples, transmitting the prompt to a first pretrained generative artificial intelligence model, receiving, from the first pretrained generative artificial intelligence model, a logical form corresponding to the natural language utterance based at least in part on the prompt, executing the logical form on a database to obtain a query result, and providing the query result to a user.
1 . A computer-implemented method comprising:
acquiring a natural language utterance and a database schema;
transmitting the natural language utterance to a generative artificial intelligence model;
receiving, from the generative artificial intelligence model, a natural language text pattern generated based on the natural language utterance;
searching, using the natural language text pattern as a key, a memory bank for one or more in-context examples that are relevant to the key, wherein the memory bank comprises in-context examples, and each of the in-context examples comprises an illustrative natural language utterance, an illustrative logical form corresponding to the illustrative natural language utterance, and an illustrative database schema;
generating a prompt comprising the natural language utterance, the database schema, and the one or more in-context examples;
transmitting the prompt to a pretrained generative artificial intelligence model;
receiving, from the pretrained generative artificial intelligence model, a logical form corresponding to the natural language utterance based at least in part on the prompt;
executing the logical form on a database to obtain a query result; and
providing the query result to a user.
2 . The computer-implemented method of claim 1 , further comprising:
receiving positive feedback from the user concerning the query result; and
responsive to receiving the positive feedback, storing the natural language utterance, the database schema, and the logical form corresponding to the natural language utterance as an additional in-context example within the memory bank.
3 . The computer-implemented method of claim 1 ,
wherein:
the natural language text pattern comprises a portion of the natural language utterance,
each of the in-context examples comprises an illustrative natural language utterance, an illustrative natural language text pattern corresponding to the illustrative natural language utterance, an illustrative logical form corresponding to the illustrative natural language utterance, and an illustrative database schema,
the memory bank is searched for the one or more in-context examples that are relevant to the key using the natural language text pattern as the key, and
the searching comprises: determining a similarity score as a metric of similarity between the key and the illustrative natural language text pattern corresponding to each of the in-context examples, comparing the similarity scores to a predetermined threshold, and identifying one or more in-context examples having a similarity score greater than or equal to the predetermined threshold as the one or more in-context examples that are relevant to the key.
4 . The computer-implemented method of claim 1 , wherein:
each of the in-context examples comprises an illustrative natural language utterance, an illustrative natural language text pattern corresponding to the illustrative natural language utterance, an illustrative logical form corresponding to the illustrative natural language utterance, and an illustrative database schema, and
the searching comprises: determining a similarity score as a metric of similarity between the key and the illustrative natural language text pattern corresponding to each of the in-context examples, comparing the similarity scores to a predetermined threshold, and identifying one or more in-context examples having a similarity score greater than or equal to the predetermined threshold as the one or more in-context examples that are relevant to the key.
5 . The computer-implemented method of claim 1 , wherein the searching comprises: determining a similarity score as a metric of similarity between the key and the illustrative natural language utterance corresponding to each of the in-context examples, comparing the similarity scores to a predetermined threshold, and identifying one or more in-context examples having a similarity score greater than or equal to the predetermined threshold as the one or more in-context examples that are relevant to the key.
6 . The computer-implemented method of claim 1 , further comprising:
comparing the database schema to the illustrative database schemas of the in-context examples;
determining, based on the comparing, the database schema is not the same as any of the illustrative database schemas; and
responsive to determining the database schema is not the same as any of the illustrative database schemas, transmitting the natural language utterance to the pretrained generative artificial intelligence model, wherein:
the natural language text pattern comprises a portion of the natural language utterance,
each of the in-context examples comprises an illustrative natural language utterance, an illustrative natural language text pattern corresponding to the illustrative natural language utterance, an illustrative logical form corresponding to the illustrative natural language utterance, and an illustrative database schema,
the memory bank is searched for the one or more in-context examples that are relevant to the key using the natural language text pattern as the key, and
the searching comprises: determining a similarity score as a metric of similarity between the key and the illustrative natural language text pattern corresponding to each of the in-context examples, comparing the similarity scores to a predetermined threshold, and identifying one or more in-context examples having a similarity score greater than or equal to the predetermined threshold as the one or more in-context examples that are relevant to the key.
7 . A system comprising:
one or more processors; and
one or more computer-readable media storing instructions which, when executed by the one or more processors, cause the system to perform operations comprising:
acquiring a natural language utterance and a database schema;
transmitting the natural language utterance to a generative artificial intelligence model;
receiving, from the generative artificial intelligence model, a natural language text pattern generated based at least in part on the natural language utterance;
searching, using the natural language text pattern as a key, a memory bank for one or more in-context examples that are relevant to the key, wherein the memory bank comprises in-context examples, and each of the in-context examples comprises an illustrative natural language utterance, an illustrative logical form corresponding to the illustrative natural language utterance, and an illustrative database schema;
generating a prompt comprising the natural language utterance, the database schema, and the one or more in-context examples;
transmitting the prompt to a pretrained generative artificial intelligence model;
receiving, from the pretrained generative artificial intelligence model, a logical form corresponding to the natural language utterance based at least in part on the prompt;
executing the logical form on a database to obtain a query result; and
providing the query result to a user.
8 . The system of claim 7 , wherein the operations further comprise:
receiving positive feedback from the user concerning the query result; and
responsive to receiving the positive feedback, storing the natural language utterance, the database schema, and the logical form corresponding to the natural language utterance as an additional in-context example within the memory bank.
9 . The system of claim 7 ,
wherein:
the natural language text pattern comprises a portion of the natural language utterance,
each of the in-context examples comprises an illustrative natural language utterance, an illustrative natural language text pattern corresponding to the illustrative natural language utterance, an illustrative logical form corresponding to the illustrative natural language utterance, and an illustrative database schema,
the memory bank is searched for the one or more in-context examples that are relevant to the key using the natural language text pattern as the key, and
the searching comprises: determining a similarity score as a metric of similarity between the key and the illustrative natural language text pattern corresponding to each of the in-context examples, comparing the similarity scores to a predetermined threshold, and identifying one or more in-context examples having a similarity score greater than or equal to the predetermined threshold as the one or more in-context examples that are relevant to the key.
10 . The system of claim 7 , wherein:
each of the in-context examples comprises an illustrative natural language utterance, an illustrative natural language text pattern corresponding to the illustrative natural language utterance, an illustrative logical form corresponding to the illustrative natural language utterance, and an illustrative database schema, and
the searching comprises: determining a similarity score as a metric of similarity between the key and the illustrative natural language text pattern corresponding to each of the in-context examples, comparing the similarity scores to a predetermined threshold, and identifying one or more in-context examples having a similarity score greater than or equal to the predetermined threshold as the one or more in-context examples that are relevant to the key.
11 . The system of claim 7 , wherein the searching comprises: determining a similarity score as a metric of similarity between the key and the illustrative natural language utterance corresponding to each of the in-context examples, comparing the similarity scores to a predetermined threshold, and identifying one or more in-context examples having a similarity score greater than or equal to the predetermined threshold as the one or more in-context examples that are relevant to the key.
12 . The system of claim 7 , wherein the operations further comprise:
comparing the database schema to the illustrative database schemas of the in-context examples;
determining, based on the comparing, the database schema is not the same as any of the illustrative database schemas; and
responsive to determining the database schema is not the same as any of the illustrative database schemas, transmitting the natural language utterance to the pretrained generative artificial intelligence model, wherein:
the natural language text pattern comprises a portion of the natural language utterance,
each of the in-context examples comprises an illustrative natural language utterance, an illustrative natural language text pattern corresponding to the illustrative natural language utterance, an illustrative logical form corresponding to the illustrative natural language utterance, and an illustrative database schema,
the memory bank is searched for the one or more in-context examples that are relevant to the key using the natural language text pattern as the key, and
the searching comprises: determining a similarity score as a metric of similarity between the key and the illustrative natural language text pattern corresponding to each of the in-context examples, comparing the similarity scores to a predetermined threshold, and identifying one or more in-context examples having a similarity score greater than or equal to the predetermined threshold as the one or more in-context examples that are relevant to the key.
13 . One or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:
acquiring a natural language utterance and a database schema;
transmitting the natural language utterance to a generative artificial intelligence model;
receiving, from the generative artificial intelligence model, a natural language text pattern generated based at least in part on the natural language utterance;
searching, using the natural language text pattern as a key, a memory bank for one or more in-context examples that are relevant to the key, wherein the memory bank comprises in-context examples, and each of the in-context examples comprises an illustrative natural language utterance, an illustrative logical form corresponding to the illustrative natural language utterance, and an illustrative database schema;
generating a prompt comprising the natural language utterance, the database schema, and the one or more in-context examples;
transmitting the prompt to a pretrained generative artificial intelligence model;
receiving, from the pretrained generative artificial intelligence model, a logical form corresponding to the natural language utterance based at least in part on the prompt;
executing the logical form on a database to obtain a query result; and
providing the query result to a user.
14 . The one or more non-transitory computer-readable media of claim 13 , wherein the operations further comprise:
receiving positive feedback from the user concerning the query result; and
responsive to receiving the positive feedback, storing the natural language utterance, the database schema, and the logical form corresponding to the natural language utterance as an additional in-context example within the memory bank.
15 . The one or more non-transitory computer-readable media of claim 13 ,
wherein:
the natural language text pattern comprises a portion of the natural language utterance,
each of the in-context examples comprises an illustrative natural language utterance, an illustrative natural language text pattern corresponding to the illustrative natural language utterance, an illustrative logical form corresponding to the illustrative natural language utterance, and an illustrative database schema,
the memory bank is searched for the one or more in-context examples that are relevant to the key using the natural language text pattern as the key, and
the searching comprises: determining a similarity score as a metric of similarity between the key and the illustrative natural language text pattern corresponding to each of the in-context examples, comparing the similarity scores to a predetermined threshold, and identifying one or more in-context examples having a similarity score greater than or equal to the predetermined threshold as the one or more in-context examples that are relevant to the key.
16 . The one or more non-transitory computer-readable media of claim 13 , wherein:
each of the in-context examples comprises an illustrative natural language utterance, an illustrative natural language text pattern corresponding to the illustrative natural language utterance, an illustrative logical form corresponding to the illustrative natural language utterance, and an illustrative database schema, and
the searching comprises: determining a similarity score as a metric of similarity between the key and the illustrative natural language text pattern corresponding to each of the in-context examples, comparing the similarity scores to a predetermined threshold, and identifying one or more in-context examples having a similarity score greater than or equal to the predetermined threshold as the one or more in-context examples that are relevant to the key.
17 . The one or more non-transitory computer-readable media of claim 13 , wherein the searching comprises: determining a similarity score as a metric of similarity between the key and the illustrative natural language utterance corresponding to each of the in-context examples, comparing the similarity scores to a predetermined threshold, and identifying one or more in-context examples having a similarity score greater than or equal to the predetermined threshold as the one or more in-context examples that are relevant to the key.
18 . The one or more non-transitory computer-readable media of claim 13 , wherein the operations further comprise:
comparing the database schema to the illustrative database schemas of the in-context examples;
determining, based on the comparing, the database schema is not the same as any of the illustrative database schemas; and
responsive to determining the database schema is not the same as any of the illustrative database schemas, transmitting the natural language utterance to the pretrained generative artificial intelligence model, wherein:
the natural language text pattern comprises a portion of the natural language utterance,
each of the in-context examples comprises an illustrative natural language utterance, an illustrative natural language text pattern corresponding to the illustrative natural language utterance, an illustrative logical form corresponding to the illustrative natural language utterance, and an illustrative database schema,
the memory bank is searched for the one or more in-context examples that are relevant to the key using the natural language text pattern as the key, and
the searching comprises: determining a similarity score as a metric of similarity between the key and the illustrative natural language text pattern corresponding to each of the in-context examples, comparing the similarity scores to a predetermined threshold, and identifying one or more in-context examples having a similarity score greater than or equal to the predetermined threshold as the one or more in-context examples that are relevant to the key.