Large language model response optimization for data pipeline generation
A system may use a large language model (“LLM”) to generate a data pipeline. The system can receive a natural language query and a selection of a plurality of data sets for generating a data pipeline and generate a prompt comprising at least: the natural language query, indications of the plurality of data sets, an indication of a format of a first computer language, and an indication of available data transformations. The system can transmit the prompt to an LLM and receive, from the LLM, a response to the prompt in the format of the first computer language. The system can parse the response in the first computer language to identify at least an indication of one or more recommended data transformations. The system can generate, based on the indication of the one or more recommended data transformations, the data pipeline using a second computer language.
1 . A computer-implemented method for generating a data pipeline using a large language model, the method comprising:
receiving, via a user interface, a natural language query and a selection of a plurality of data sets for generating a data pipeline in a first computer language, wherein the first computer language is a typed computer language, and wherein a description of a data transformation in the first computer language comprises at least one of a typed expression or typed variable;
generating a prompt comprising at least: the natural language query, indications of the plurality of data sets, an indication of a format of a second computer language, and an indication of available data transformations, wherein the available data transformations comprise the data transformation, and wherein a second description of the data transformation in the second computer language is typeless;
transmitting the prompt to a large language model (“LLM”);
receiving, from the LLM, a response to the prompt in the format of the second computer language;
parsing the response in the second computer language to identify at least an indication of one or more recommended data transformations to be performed in the data pipeline; and
generating, based on the indication of the one or more recommended data transformations, the data pipeline using the first computer language.
2 . The method of claim 1 , wherein the user interface comprises a graphical user interface, and wherein the natural language prompt and the selection of a plurality of data sets are received via the graphical user interface.
3 . The method of claim 2 , wherein receiving the selection of the plurality of data sets comprises receiving, via the graphical user interface, user selection of indications of the plurality of data sets.
4 . The method of claim 3 , further comprising generating, in response to the user selection of the indications of the plurality of data sets, a prompt selector within the graphical user interface.
5 . The method of claim 4 , wherein receiving the natural language query comprises receiving user input via the prompt selector.
6 . The method of claim 1 , further comprising:
receiving, via the user interface, a second user selection of at least one of the plurality of data sets and of a second natural language prompt related to a function of at least one of the one or more recommended data transformations associated with the data pipeline;
transmitting at least a portion of second query to the LLM, the second query comprising the second natural language prompt and an indication of the at least one of the plurality of data sets;
receiving, from the LLM, a second response to the at least the portion of the second query; and
generating, based on the second response, an indication of the function of the at least one of the one or more recommended data transformations associated with the data pipeline.
7 . The method of claim 6 , wherein generating the indication of the function of the at least one of the one or more recommended transforms associated with the data pipeline comprises displaying, within the user interface, the indication of the function of the at least one of the one or more transforms.
8 . The method of claim 1 , wherein the indication of the format of the second computer language and the indication of available data transformations are received via the user interface.
9 . The method of claim 1 , wherein the indication of the format of the second computer language comprises an indication of a plurality of acceptable natural language terms.
10 . The method of claim 9 , wherein each of the plurality of acceptable natural language terms comprise a common root term each having a different spelling, a different capitalization, and/or a different punctuation from another of the plurality of acceptable natural language terms.
11 . The method of claim 1 , wherein the indication of the format comprises an indication of a grammar of the first computer language.
12 . The method of claim 1 , wherein parsing the response in the second computer language further comprises identifying an indication of a new data set, and wherein the data pipeline comprises the new data set.
13 . The method of claim 12 , wherein generating the data pipeline comprises displaying, within the user interface, indications of:
nodes each corresponding to respective of the plurality of data sets and the new data set; and
at least one connector between at least one of the plurality of data sets and the new data set.
14 . The method of claim 12 , further comprising generating, based on the parsing of the response and the identified second query, a second data pipeline using the second computer language, the second data pipeline comprising the new data set.
15 . The method of claim 13 , further comprising parsing the response in the second computer language to further identify: an indication of a second one or more recommended data transformations and an indication of a second new data set.
16 . The method of claim 12 , further comprising:
displaying, within the user interface, the data pipeline comprising the new data set.
17 . The method of claim 1 , further comprising:
displaying, within the user interface, indications of the plurality of data sets, wherein receiving the selection of the plurality of data sets comprises receiving user selection of graphical representations of the plurality of data sets.
18 . The method of claim 1 , wherein:
the indication of the available data transformations comprises at least one of: a textual list of the available data transformations, one or more explanations of the available data transformations, one or more examples of the available data transformations, or formatting of the available data transformations according to the second computer language;
the indication of the format of the second computer language comprises at least one of: a grammar, a list of available terms, examples of acceptable instructions, or a schema for responding to the prompt; and
the prompt further comprises an indication of a schema.
19 . A system comprising:
one or more computer-readable storage mediums comprising program instructions; and
one or more processors configured to execute the program instructions to cause the system to perform the computer-implemented method of claim 1 .
20 . One or more non-transitory computer-readable storage mediums comprising program instructions, the program instructions executable by one or more processors to cause the one or more processors to perform the computer-implemented method of claim 1 .