Automated metadata generation and natural language database querying
A machine-readable medium stores instructions that, when executed by a processor, cause the processor to execute operations. The operations include receiving information identifying a database table and preprocessing the table. The preprocessing includes extracting primary and foreign key relationships and generating sample data by combining frequency-based sampling of common values with random sampling. The operations also include generating, using an LLM, a unified metadata schema. The generating includes processing the table and the sample data using domain-specific prompts and outputting the unified metadata schema. The operations include embedding and indexing the unified metadata schema and converting a natural language question output from an end-user device into a SQL query. The converting includes identifying relevant metadata in the unified metadata schema matching query intent and generating the SQL query using the filtered metadata. The operations also include executing the SQL query and providing query results to the end-user device.
1 . A non-transitory machine-readable medium storing instructions that, when executed by a processor, cause the processor to execute operations, the operations comprising:
receiving information identifying a database table;
preprocessing the database table, the preprocessing comprising:
extracting primary and foreign key relationships; and
generating sample data by combining frequency-based sampling of most common values with random sampling;
generating, using an LLM (large language model), a unified metadata schema, the generating comprising:
processing the preprocessed database table and the sample data using domain-specific prompts; and
outputting the unified metadata schema in a standardized format;
embedding and indexing the unified metadata schema;
converting a natural language question output from an end-user device into a SQL (structured query language) query, the converting comprising:
identifying relevant metadata in the unified metadata schema matching query intent using semantic search;
filtering and prioritizing metadata based on query relevance; and
generating the SQL query using the filtered metadata;
executing the SQL query against the database; and
providing query results to the end-user device as a response to the natural language question.
2 . The non-transitory machine-readable medium of claim 1 , wherein the operations further comprise logging user feedback and query execution operations to enable improvement in generating the SQL query.
3 . The non-transitory machine-readable medium of claim 1 , wherein preprocessing the database table comprises applying an elbow method to identify a point of diminishing returns in data completeness at a threshold percentage of non-null values and removing columns having null values meeting or exceeding the threshold percentage.
4 . The non-transitory machine-readable medium of claim 1 , wherein generating the sample data comprises collecting data points for each column by combining most frequently occurring values with random samples.
5 . The non-transitory machine-readable medium of claim 1 , wherein the domain-specific prompts incorporate industry and database context information to improve metadata generation accuracy.
6 . The non-transitory machine-readable medium of claim 1 , wherein the unified metadata schema comprises:
table name and description;
column names, data types and descriptions;
key terms and sample data; and
table relationships and constraints.
7 . The non-transitory machine-readable medium of claim 1 , wherein embedding the metadata comprises creating vector representations to enable semantic search capabilities.
8 . The non-transitory machine-readable medium of claim 1 , wherein converting the natural language question comprises validating query accuracy using metadata context and returning error messages rather than hallucinating responses when unable to generate a valid SQL query.
9 . The non-transitory machine-readable medium of claim 1 , wherein the operations further comprise processing one database table at a time to curtail hallucinations.
10 . A system for automated metadata generation and natural language database querying, comprising:
a metadata generation module operating on one or more computing platforms that:
receives information identifying a database table;
preprocesses the database table by extracting primary and foreign key relationships;
generates balanced sample data by combining frequency-based sampling of most common values with random sampling; and
generates, using an LLM (large language model), a unified metadata schema using domain-specific prompts and outputs the unified metadata schema in a standardized format;
a retrieval augmented generation module operating on the one or more computing platforms that:
embeds and indexes the unified metadata schema;
identifies relevant metadata in the unified metadata schema matching user intent of a question using a semantic search; and
filters and prioritizes metadata based on query relevance;
a data retrieval agent operating on the one or more computing platforms that:
receives a natural language question from an end-user device; and
converts the natural language question into a SQL (structured query language) query using the filtered metadata;
executes the SQL query against the database; and
provides query results to the end-user device as a response to the natural language question.
11 . The system of claim 10 , wherein preprocessing the database table comprises applying an elbow method to identify a point of diminishing returns in data completeness at a threshold percentage of non-null values and removing columns having null values meeting or exceeding the threshold percentage.
12 . The system of claim 10 , wherein generating the balanced sample data comprises data points for each column by combining most frequently occurring values with random samples.
13 . The system of claim 10 , wherein the unified metadata schema comprises table name and description, column names and data types, key terms and sample data and table relationships and constraints.
14 . The system of claim 10 , wherein the metadata generation module processes one database table at a time to avoid hallucinations.
15 . The system of claim 10 , wherein the data retrieval agent validates query accuracy using metadata context and returns an error when unable to generate a valid SQL query.
16 . The system of claim 10 , wherein the data retrieval agent further logs user feedback and query execution operation, wherein the user feedback characterizes approval or disapproval ratings for answers to natural language questions.
17 . The system of claim 10 , wherein the domain-specific prompts incorporate industry and database context information to improve metadata generation accuracy.
18 . A method for automated metadata generation and natural language database querying, comprising:
preprocessing, by a metadata generation module operating on one or more computing platforms, a database table, the preprocessing comprising:
extracting primary and foreign key relationships; and
generating sample data by combining frequency-based sampling of most common values with random sampling; and
generating, using an LLM (large language model), a unified metadata schema, wherein the generating comprises:
processing the preprocessed database table and the sample data using domain-specific prompts; and
outputting the unified metadata schema in a standardized format;
embedding and indexing the unified metadata schema with a retrieval augmented generation module operating on the one or more computing platforms;
converting, by a data retrieval agent operating on the one or more computing platforms a natural language question provided from an end-user device into an SQL (structured query language) query, wherein the converting comprises:
identifying relevant metadata in the unified metadata schema matching query intent using semantic search;
filtering and prioritizing metadata based on query relevance; and
generating the SQL query using the filtered metadata;
executing, by the data retrieval agent the SQL query against the database; and
outputting, by the data retrieval agent, query results to the end-user device as a response to the natural language question.
19 . The method of claim 18 , wherein preprocessing the database table comprises:
analyzing data completeness patterns using an elbow method to identify a set of columns that have at least a threshold percentage of null values;
removing the set of columns that having null values meeting or exceeding the threshold percentage; and
collecting data points for each retained column by combining most frequently occurring values with random samples.
20 . The method of claim 18 , wherein generating the unified metadata schema comprises:
incorporating table name, description and business function;
documenting column names, data types, descriptions and constraints;
specifying relationships between tables and foreign key connections;
storing the metadata in JSON format to support interoperability across platforms; and
processing one database table at a time to curtail hallucination.