Priming generative AI model leveraging directed acyclic graph-driven notebook environment
An application receives a natural language query from a user into a cell of a notebook environment and responsively determines a set of precedent cells and a profile of the user. The application determines a portion of the data warehouse graph that corresponds to the natural language query. The application primes the large language model with priming context that is based on the portion of the data warehouse graph that corresponds to the natural language query, the precedent cells from which the code cell depends, and the profile of the user, the priming resulting in a primed large language model. The application inputs the natural language query into the primed large language model and receives, as output from the large language model, a response to the natural language query. The application provides the response to the natural language query to the user.
1. A method comprising:
receiving a natural language query from a user into a cell of a notebook environment, the natural language query performed with respect to a data warehouse, the data warehouse modeled in a data warehouse graph;
in response to receiving the natural language query:
determining, using directed edges of a notebook graph structure, a set of precedent cells from which the cell depends, the notebook graph structure being a directed acyclic graph; and
determining, using edges of a user graph structure populated based on activities of users relative to the notebook environment, a profile of the user;
determining a portion of the data warehouse graph that corresponds to the natural language query;
priming a large language model with priming context, the priming context based on the portion of the data warehouse graph that corresponds to the natural language query, the precedent cells from which the cell depends, and the profile of the user, the priming resulting in a primed large language model;
inputting the natural language query into the primed large language model;
receiving, as output from the large language model, a response to the natural language query; and
providing the response to the natural language query to the user.
2. The method of claim 1 , wherein the data warehouse is modeled by:
generating a column node for each column of each table within the data warehouse; and
generating edges between column nodes that are mutually queried.
3. The method of claim 2 , wherein column nodes are mutually queried when they are called together by a cell.
4. The method of claim 2 , wherein column nodes are mutually queried when they are together part of a join command.
5. The method of claim 2 , wherein the edges between the column nodes indicate information about how they were mutually queried.
6. The method of claim 5 , wherein the information is pointed to by one or more nodes of the user graph structure for given users that performed an act in which the column nodes were mutually queried.
7. The method of claim 1 , wherein determining the set of precedent cells from which the cell depends comprises determining values from the set of precedent cells using a sequential ordering dictated by directed edges of the notebook graph structure.
8. The method of claim 1 , wherein determining the profile of the user comprises determining historical queries in historical projects performed by the user that relate to the natural language query.
9. A non-transitory computer-readable medium comprising memory with instructions encoded thereon that, when executed, cause one or more processors to perform operations, the instructions comprising instructions to:
receive a natural language query from a user into a cell of a notebook environment, the natural language query performed with respect to a data warehouse, the data warehouse modeled in a data warehouse graph;
in response to receiving the natural language query:
determine, using directed edges of a notebook graph structure, a set of precedent cells from which the cell depends, the notebook graph structure being a directed acyclic graph; and
determine, using edges of a user graph structure populated based on activities of users relative to the notebook environment, a profile of the user;
determine a portion of the data warehouse graph that corresponds to the natural language query;
prime a large language model with priming context, the priming context based on the portion of the data warehouse graph that corresponds to the natural language query, the precedent cells from which the cell depends, and the profile of the user, the priming resulting in a primed large language model;
input the natural language query into the primed large language model;
receive, as output from the large language model, a response to the natural language query; and
provide the response to the natural language query to the user.
10. The non-transitory computer-readable medium of claim 9 , wherein the data warehouse is modeled by:
generating a column node for each column of each table within the data warehouse; and
generating edges between column nodes that are mutually queried.
11. The non-transitory computer-readable medium of claim 10 , wherein column nodes are mutually queried when they are called together by a cell.
12. The non-transitory computer-readable medium of claim 10 , wherein column nodes are mutually queried when they are together part of a join command.
13. The non-transitory computer-readable medium of claim 10 , wherein the edges between the column nodes indicate information about how they were mutually queried.
14. The non-transitory computer-readable medium of claim 13 , wherein the information is pointed to by one or more nodes of the user graph structure for given users that performed an act in which the column nodes were mutually queried.
15. The non-transitory computer-readable medium of claim 9 , wherein the instructions to determine the set of precedent cells from which the cell depends comprise instructions to determine values from the set of precedent cells using a sequential ordering dictated by directed edges of the notebook graph structure.
16. The non-transitory computer-readable medium of claim 9 , wherein the instructions to determine the profile of the user comprise instructions to determine historical queries in historical projects performed by the user that relate to the natural language query.
17. A system comprising:
memory with instructions encoded thereon; and
one or more processors that, when executing the instructions, are caused to perform operations comprising:
receiving a natural language query from a user into a cell of a notebook environment, the natural language query performed with respect to a data warehouse, the data warehouse modeled in a data warehouse graph;
in response to receiving the natural language query:
determining, using directed edges of a notebook graph structure, a set of precedent cells from which the cell depends, the notebook graph structure being a directed acyclic graph; and
determining, using edges of a user graph structure populated based on activities of users relative to the notebook environment, a profile of the user;
determining a portion of the data warehouse graph that corresponds to the natural language query;
priming a large language model with priming context, the priming context based on the portion of the data warehouse graph that corresponds to the natural language query, the precedent cells from which the cell depends, and the profile of the user, the priming resulting in a primed large language model;
inputting the natural language query into the primed large language model;
receiving, as output from the large language model, a response to the natural language query; and
providing the response to the natural language query to the user.
18. The system of claim 17 , wherein the data warehouse is modeled by:
generating a column node for each column of each table within the data warehouse; and
generating edges between column nodes that are mutually queried.
19. The system of claim 18 , wherein column nodes are mutually queried when they are called together by a cell.
20. The system of claim 18 , wherein column nodes are mutually queried when they are together part of a join command.