AI agent-based semantic data updater
Various example embodiments described herein provide for systems, methods, devices, instructions, and like for translation to semantic data for artificial intelligence (AI) agent-based semantic data updates (e.g., improvements) using one or more data sources, such as golden data sources, where the updated semantic data (e.g., semantic model) that results can be used by a natural language query system to translate one or more natural language questions into one or more structure language queries (e.g., one or more structured query language (SQL) statements). Some example embodiments are implemented as a system that uses a set of AI agents working in an optimization loop to improve/refine semantic data using the one or more data sources.
1 . A system comprising:
at least one hardware processor; and
at least one memory storing instructions that cause the at least one hardware processor to perform operations comprising:
accessing a baseline semantic model that describes a schema of a database and that enables a large language model (LLM) to generate structured language data queries for the database based on natural language questions;
accessing a data source that comprises a set of verified pairs of natural language questions and structured language data queries; and
causing execution of an orchestrator artificial intelligence (AI) agent that coordinates a plurality of individual AI agents to update the baseline semantic model with one or more improvements, the plurality of individual AI agents comprising:
a relationship suggestion AI agent that receives a select version of the baseline semantic model from the orchestrator AI agent, determines one or more table relationship suggestions for the select version of the baseline semantic model based on analyzing one or more structured language data query patterns in the set of verified pairs, and updates the select version of the baseline semantic model based on the one or more table relationship suggestions;
a search suggestion AI agent that receives a given version of the baseline semantic model from the orchestrator AI agent, determines one or more searchable dimension suggestions for the given version of the baseline semantic model based on dimension cardinality, and updates the given version of the baseline semantic model based on the one or more searchable dimension suggestions;
an edit suggestion AI agent that receives an individual version of the baseline semantic model from the orchestrator AI agent, determines one or more edit suggestions for the individual version of the baseline semantic model to improve accuracy of the individual version of the baseline semantic model, and updates the individual version of the baseline semantic model; and
a validation AI agent that receives a subsequent version of the baseline semantic model, and performs an evaluation process on the subsequent version of the baseline semantic model to analyze and validate one or more updates applied to the subsequent version of the baseline semantic model.
2 . The system of claim 1 , wherein at least one of the orchestrator AI agent, the relationship suggestion AI agent, the search suggestion AI agent, the edit suggestion AI agent, or the validation AI agent comprises a LLM agent.
3 . The system of claim 1 , wherein the relationship suggestion AI agent determines the one or more table relationship suggestions for the select version of the baseline semantic model based on the analyzing of the one or more structured language data query patterns in the set of verified pairs by:
generating a prompt based on the baseline semantic model and the set of verified pairs, the prompt comprising one or more instructions for determining the one or more table relationship suggestions for the select version of the baseline semantic model based on the analyzing of the one or more structured language data query patterns in the set of verified pairs; and
generating, by a select LLM, an output based on the prompt, at least one table relationship suggestion of the one or more table relationship suggestions being determined based on the output.
4 . The system of claim 3 , wherein the one or more instructions of the prompt comprise at least one of:
a first instruction to identify one or more join paths in at least one structured language data query in the set of verified pairs;
a second instruction to deduplicate the one or more join paths; or
a third instruction to generate, as part of the output, one or more table relationship definitions identified based on the deduplicated one or more join paths.
5 . The system of claim 1 , wherein the search suggestion AI agent determines the one or more searchable dimension suggestions for the given version of the baseline semantic model based on dimension cardinality by:
determining a set of approximate cardinalities for a set of dimensions within the given version of the baseline semantic model;
filtering the set of dimensions based on the set of approximate cardinalities to generate a filtered set of dimensions;
based on one or more pairs of the set of verified pairs, generating a first prompt that comprises one or more instructions to determine, for each individual pair of the one or more pairs, one or more dimensions of the filtered set of dimensions that help answer an individual natural language question of the individual pair;
generating, by a select LLM, a first output based on the first prompt;
generating a second prompt that comprises one or more instructions to aggregate one or more results from the first output and generate a second output that comprises a list of columns; and
generating, by the select LLM, the second output based on the second prompt, at least one searchable dimension suggestion of the one or more searchable dimension suggestions being determined based on the list of columns from the second output.
6 . The system of claim 1 , wherein the edit suggestion AI agent determines one or more edit suggestions for the individual version of the baseline semantic model to improve accuracy of the individual version of the baseline semantic model by:
generating a prompt that comprises one or more instructions to analyze the individual version of the baseline semantic model for one or more discrepancies and determine one or more proposed edits to the individual version of the baseline semantic model based on a resulting analysis; and
generating, by a select LLM, an output based on the prompt, at least one edit suggestion of the one or more edit suggestions being determined based on the one or more proposed edits from the output.
7 . The system of claim 1 , wherein the one or more edit suggestions comprise at least one of:
an addition of a dimension to the individual version of the baseline semantic model;
an addition of a fact to the individual version of the baseline semantic model;
an addition of a metric to the individual version of the baseline semantic model;
an addition of a filter to the individual version of the baseline semantic model;
a name change in the individual version of the baseline semantic model;
a description change in the individual version of the baseline semantic model;
a synonym change in the individual version of the baseline semantic model; or
an expression change in the individual version of the baseline semantic model.
8 . The system of claim 1 , wherein the edit suggestion AI agent determines one or more edit suggestions for the individual version of the baseline semantic model to improve accuracy of the individual version of the baseline semantic model by:
based on one or more existing custom instructions from the individual version of the baseline semantic model, generating a prompt that comprises one or more instructions to update the one or more existing custom instructions; and
generating, by a select LLM, an output based on the prompt, the output comprising at least one updated custom instruction, at least one edit suggestion of the one or more edit suggestions comprising updating at least one existing custom instruction in the individual version of the baseline semantic model based on the at least one updated custom instruction.
9 . The system of claim 1 , wherein the evaluation process performed by the validation AI agent performs comprises:
based on the subsequent version of the baseline semantic model, generating, by an individual LLM, an example structured language data query for a select natural language question from an individual verified pair of the set of verified pairs;
determining whether a set of columns of the example structured language data query are different from a set of columns of a correct structured language data query from the individual verified pair; and
in response to determining that the set of columns of the example structured language data query are not different from the set of columns of the correct structured language data query, indicating that the example structured language data query is valid.
10 . The system of claim 9 , wherein the evaluation process performed by the validation AI agent performs comprises:
in response to determining that the set of columns of the example structured language data query are different from the set of columns of the correct structured language data query:
generating a first query result from the database based on the example structured language data query;
generating a second query result from the database based on a correct structured language data query from the individual verified pair;
using a select LLM to determine whether the first query result and the second query result are equivalent; and
in response to determining that the first query result and the second query result are equivalent, indicating that the example structured language data query is valid.
11 . The system of claim 1 , wherein the validation AI agent tracks an accuracy improvement achieved by an individual AI agent of the plurality of AI agents after the individual AI agent updates a latest version of the baseline semantic model.
12 . The system of claim 1 , wherein the orchestrator AI agent coordinates the plurality of individual AI agents to update the baseline semantic model with the one or more improvements using an iterative loop, wherein the orchestrator AI agent selects which one or more AI agents of the plurality of AI agents are used to update the baseline semantic model during an individual iteration of the iterative loop, and wherein the orchestrator AI agent selects an order in which the one or more AI agents are used to update the baseline semantic model during the individual iteration of the iterative loop.
13 . The system of claim 12 , wherein the orchestrator AI agent determines how many iterations of the iterative loop are performed.
14 . A method comprising:
accessing, by at least one processor, a baseline semantic model that describes a schema of a database and that enables a large language model (LLM) to generate structured language data queries for the database based on natural language questions;
accessing, by the at least one processor, a data source that comprises a set of verified pairs of natural language questions and structured language data queries; and
causing, by the at least one processor, execution of an orchestrator artificial intelligence (AI) agent that coordinates a plurality of individual AI agents to update the baseline semantic model with one or more improvements, the plurality of individual AI agents comprising:
a relationship suggestion AI agent that receives a select version of the baseline semantic model from the orchestrator AI agent, determines one or more table relationship suggestions for the select version of the baseline semantic model based on analyzing one or more structured language data query patterns in the set of verified pairs, and updates the select version of the baseline semantic model based on the one or more table relationship suggestions;
a search suggestion AI agent that receives a given version of the baseline semantic model from the orchestrator AI agent, determines one or more searchable dimension suggestions for the given version of the baseline semantic model based on dimension cardinality, and updates the given version of the baseline semantic model based on the one or more searchable dimension suggestions;
an edit suggestion AI agent that receives an individual version of the baseline semantic model from the orchestrator AI agent, determines one or more edit suggestions for the individual version of the baseline semantic model to improve accuracy of the individual version of the baseline semantic model, and updates the individual version of the baseline semantic model; and
a validation AI agent that receives a subsequent version of the baseline semantic model, and performs an evaluation process on the subsequent version of the baseline semantic model to analyze and validate one or more updates applied to the subsequent version of the baseline semantic model.
15 . The method of claim 14 , wherein at least one of the orchestrator AI agent, the relationship suggestion AI agent, the search suggestion AI agent, the edit suggestion AI agent, or the validation AI agent comprises a LLM agent.
16 . The method of claim 14 , wherein the relationship suggestion AI agent determines the one or more table relationship suggestions for the select version of the baseline semantic model based on the analyzing of the one or more structured language data query patterns in the set of verified pairs by:
generating a prompt based on the baseline semantic model and the set of verified pairs, the prompt comprising one or more instructions for determining the one or more table relationship suggestions for the select version of the baseline semantic model based on the analyzing of the one or more structured language data query patterns in the set of verified pairs; and
generating, by a select LLM, an output based on the prompt, at least one table relationship suggestion of the one or more table relationship suggestions being determined based on the output.
17 . The method of claim 14 , wherein the search suggestion AI agent determines the one or more searchable dimension suggestions for the given version of the baseline semantic model based on dimension cardinality by:
determining a set of approximate cardinalities for a set of dimensions within the given version of the baseline semantic model;
filtering the set of dimensions based on the set of approximate cardinalities to generate a filtered set of dimensions;
based on one or more pairs of the set of verified pairs, generating a first prompt that comprises one or more instructions to determine, for each individual pair of the one or more pairs, one or more dimensions of the filtered set of dimensions that help answer an individual natural language question of the individual pair;
generating, by a select LLM, a first output based on the first prompt;
generating a second prompt that comprises one or more instructions to aggregate one or more results from the first output generated and generate a second output that comprises a list of columns; and
generating, by the select LLM, the second output based on the second prompt, at least one searchable dimension suggestion of the one or more searchable dimension suggestions being determined based on the list of columns from the second output.
18 . The method of claim 14 , wherein the edit suggestion AI agent determines one or more edit suggestions for the individual version of the baseline semantic model to improve accuracy of the individual version of the baseline semantic model by:
generating a prompt that comprises one or more instructions to analyze the individual version of the baseline semantic model for one or more discrepancies and determine one or more proposed edits to the individual version of the baseline semantic model based on a resulting analysis; and
generating, by a select LLM, an output based on the prompt, at least one edit suggestion of the one or more edit suggestions being determined based on the one or more proposed edits from the output.
19 . The method of claim 14 , wherein the edit suggestion AI agent determines one or more edit suggestions for the individual version of the baseline semantic model to improve accuracy of the individual version of the baseline semantic model by:
based on one or more existing custom instructions from the individual version of the baseline semantic model, generating a prompt that comprises one or more instructions to update the one or more existing custom instructions; and
generating, by a select LLM, an output based on the prompt, the output comprising at least one updated custom instruction, at least one edit suggestion of the one or more edit suggestions comprising updating at least one existing custom instruction in the individual version of the baseline semantic model based on the at least one updated custom instruction.
20 . A machine-storage medium, the machine-storage medium including instructions that when executed by a machine, cause the machine to perform operations comprising:
accessing a baseline semantic model that describes a schema of a database and that enables a large language model (LLM) to generate structured language data queries for the database based on natural language questions;
accessing a data source that comprises a set of verified pairs of natural language questions and structured language data queries; and
causing execution of an orchestrator artificial intelligence (AI) agent that coordinates a plurality of individual AI agents to update the baseline semantic model with one or more improvements, the plurality of individual AI agents comprising:
a relationship suggestion AI agent that receives a select version of the baseline semantic model from the orchestrator AI agent, determines one or more table relationship suggestions for the select version of the baseline semantic model based on analyzing one or more structured language data query patterns in the set of verified pairs, and updates the select version of the baseline semantic model based on the one or more table relationship suggestions;
a search suggestion AI agent that receives a given version of the baseline semantic model from the orchestrator AI agent, determines one or more searchable dimension suggestions for the given version of the baseline semantic model based on dimension cardinality, and updates the given version of the baseline semantic model based on the one or more searchable dimension suggestions;
an edit suggestion AI agent that receives an individual version of the baseline semantic model from the orchestrator AI agent, determines one or more edit suggestions for the individual version of the baseline semantic model to improve accuracy of the individual version of the baseline semantic model, and updates the individual version of the baseline semantic model; and
a validation AI agent that receives a subsequent version of the baseline semantic model, and performs an evaluation process on the subsequent version of the baseline semantic model to analyze and validate one or more updates applied to the subsequent version of the baseline semantic model.