Detection of links among tabular data
Detection of links among tabular data is disclosed, including: obtaining a set of source data; extract portions from the set of source data to store in cells within a source dataset according to a source schema determined for the set of source data; determining a link between a first data value stored in a first cell of the source dataset to a second data value stored in a second cell of a destination dataset of stored tabular data; and in response to a search that matches the first data value stored in the first cell, returning the first data value stored in the first cell and the second data value stored in the second cell based at least in part on the link.
1 . A system, comprising:
a storage device configured to store tabular data comprising a plurality of tables; and
one or more processors configured to:
obtain a set of source data;
extract portions from the set of source data to store in cells within a source dataset according to a source schema determined for the set of source data, wherein the source dataset comprises a source table;
determine a link between a first data value stored in a first cell of the source dataset to a second data value stored in a second cell of a destination dataset of stored tabular data, wherein the destination dataset comprises a destination table, wherein the first cell is included in a first row of the source dataset, wherein the first data value comprises a first set of global positioning data (GPS) data, wherein the second cell is included in a second row of the destination dataset, wherein the second data value comprises a second set of GPS data, wherein to determine the link between the first data value stored in the first cell of the source dataset to the second data value stored in the second cell of the destination dataset of the stored tabular data comprises to:
compare the first set of GPS data to the second set of GPS data to determine a proximity; and
in response to a determination that the proximity is less than a threshold, determine that the link is detected between the first row of the source dataset and the second row of the destination dataset, wherein the link comprises a geographical match type of link;
store data associated with the link in a link index, wherein the data associated with the link describes the first cell of the source dataset and the second cell of the destination dataset;
obtain a tabular data-specific query language query;
search at least the source dataset using the tabular data-specific query language query; and
in response to a determination that the tabular data-specific query language query matches the first data value stored in the first cell of the source dataset, return the first data value stored in the first cell of the source dataset and the second data value stored in the second cell of the destination dataset based at least in part on the link stored in the link index.
2 . The system of claim 1 , wherein to extract the portions from the set of source data to store in the cells within the source dataset according to the source schema determined for the set of source data comprises to:
receive user provided target column definitions;
generate a prompt to a large language model (LLM) to cause the LLM to analyze the set of source data to infer candidate column names;
determine a normalized schema for the source table corresponding to the set of source data based on mappings between the user provided target column definitions and the candidate column names obtained from the LLM, wherein the source schema comprises the normalized schema; and
populate the source table with rows of data extracted from the set of source data in accordance with the normalized schema.
3 . The system of claim 1 , wherein the one or more processors are further configured to:
obtain enrichment information corresponding to one or more data values stored in a third row in at least one existing column of the source schema; and
add the enrichment information into the third row in a new column of the source dataset.
4 . The system of claim 1 , wherein the one or more processors are further configured to determine a second link between a third data value stored in a third cell of the source dataset to a fourth data value stored in a fourth cell of the destination dataset of the stored tabular data comprises to:
use a hash table to map the third data value to a corresponding bucket; and
in response to a determination that the fourth data value is also stored in the corresponding bucket, determine that the second link is detected between the third data value in the third cell of the source dataset and the fourth data value in the fourth cell of the destination dataset, wherein the second link comprises an exact match type of link.
5 . The system of claim 1 , wherein the one or more processors are further configured to determine a second link between a third data value stored in a third cell of the source dataset to a fourth data value stored in a fourth cell of the destination dataset of the stored tabular data comprises to:
determine a difference between the third data value stored in the third cell of the source dataset and the fourth data value stored in the fourth cell of the destination dataset of the stored tabular data; and
in response to a determination that the difference is less than another threshold, determine that the second link is detected between the third data value in the third cell of the source dataset and the fourth data value in the fourth cell of the destination dataset, wherein the second link comprises a fuzzy match type of link.
6 . The system of claim 5 , wherein the difference between the third data value stored in the third cell of the source dataset and the fourth data value stored in the fourth cell of the destination dataset of the stored tabular data comprises an edit distance.
7 . The system of claim 1 , wherein the one or more processors are further configured to determine a second link between a third data value stored in a third cell of the source dataset to a fourth data value stored in a fourth cell of the destination dataset of the stored tabular data comprises to:
determine a first vector based at least in part on a first set of data values stored in cells in a third row of the source dataset, wherein the third row includes the third cell;
determine a second vector based at least in part on a second set of data values stored in cells in a fourth row of the destination dataset, wherein the fourth row includes the fourth cell;
compare the first vector against the second vector to determine a similarity; and
in response to a determination that the similarity is greater than another threshold, determine that the second link is detected between the third row of the source dataset and the fourth row of the destination dataset, wherein the second link comprises a vector match type of link.
8 . The system of claim 1 , wherein the proximity comprises a first proximity, wherein the threshold comprises a first threshold, wherein a third cell is included in a third row of the source dataset and includes a third data value that comprises a first set of taxonomy data, wherein a fourth cell is included in a fourth row of the destination dataset and includes a fourth data value that comprises a second set of taxonomy data, wherein the one or more processors are further configured to determine a second link between the third data value stored in the third cell of the source dataset to the fourth data value stored in the fourth cell of the destination dataset of the stored tabular data comprises to:
compare the first set of taxonomy data to the second set of taxonomy data to determine a second proximity; and
in response to a determination that the second proximity is less than a second threshold, determine that the second link is detected between the third row of the source dataset and the fourth row of the destination dataset, wherein the second link comprises a related entity type of link.
9 . The system of claim 1 , wherein to obtain the tabular data-specific query language query comprises to:
receive, via a user interface, a user submitted question;
generate a prompt to a large language model (LLM), wherein the prompt is configured to cause the LLM to generate the tabular data-specific query language query based at least in part on the user submitted question;
send the prompt to the LLM; and
receive the tabular data-specific query language query from the LLM.
10 . The system of claim 1 , wherein the one or more processors are further configured to return a link type associated with the link.
11 . The system of claim 1 , wherein the destination dataset comprises a first destination dataset, wherein the link comprises a first link, and wherein the one or more processors are further configured to:
determine whether to hide a second link between the first data value stored in the first cell and a third data value stored in a third cell in a second destination dataset; and
in response to a determination to hide the second link, omit returning information on the second link and the third data value.
12 . The system of claim 1 , wherein to return the first data value stored in the first cell and the second data value stored in the second cell based at least in part on the link comprises to:
format the first data value, the link, and the second data value in a string representation; and
generate a prompt to a large language model (LLM) that includes a user submitted question and the string representation of the first data value, the link, and the second data value.
13 . A method, comprising:
obtaining a set of source data;
extracting portions from the set of source data to store in cells within a source dataset according to a source schema determined for the set of source data, wherein the source dataset comprises a source table;
determining a link between a first data value stored in a first cell of the source dataset to a second data value stored in a second cell of a destination dataset of stored tabular data, wherein the destination dataset comprises a destination table, wherein the first cell is included in a first row of the source dataset, wherein the first data value comprises a first set of global positioning data (GPS) data, wherein the second cell is included in a second row of the destination dataset, wherein the second data value comprises a second set of GPS data, wherein determining the link between the first data value stored in the first cell of the source dataset to the second data value stored in the second cell of the destination dataset of the stored tabular data comprises:
comparing the first set of GPS data to the second set of GPS data to determine a proximity; and
in response to a determination that the proximity is less than a threshold, determining that the link is detected between the first row of the source dataset and the second row of the destination dataset, wherein the link comprises a geographical match type of link;
storing data associated with the link in a link index, wherein the data associated with the link describes the first cell of the source dataset and the second cell of the destination dataset;
obtaining a tabular data-specific query language query;
searching at least the source dataset using the tabular data-specific query language query; and
in response to a determination that the tabular data-specific query language query matches the first data value stored in the first cell of the source dataset, returning the first data value stored in the first cell of the source dataset and the second data value stored in the second cell of the destination dataset based at least in part on the link stored in the link index.
14 . The method of claim 13 , wherein extracting the portions from the set of source data to store in the cells within the source dataset according to the source schema determined for the set of source data comprises:
receiving user provided target column definitions;
generating a prompt to a large language model (LLM) to cause the LLM to analyze the set of source data to infer candidate column names;
determining a normalized schema for a new source table corresponding to the set of source data based on mappings between the user provided target column definitions and the candidate column names obtained from the LLM, wherein the source schema comprises the normalized schema; and
populating the source table with rows of data extracted from the set of source data in accordance with the normalized schema.
15 . The method of claim 13 , further comprising:
obtaining enrichment information corresponding to one or more data values stored in a third row in at least one existing column of the source schema; and
adding the enrichment information into the third row in a new column of the source dataset.
16 . The method of claim 13 , further comprising determining a second link between a third data value stored in a third cell of the source dataset to a fourth data value stored in a fourth cell of the destination dataset of the stored tabular data comprises:
using a hash table to map the third data value to a corresponding bucket; and
in response to a determination that the fourth data value is also stored in the corresponding bucket, determining that the second link is detected between the third data value in the third cell of the source dataset and the fourth data value in the fourth cell of the destination dataset, wherein the second link comprises an exact match type of link.
17 . The method of claim 13 , further comprising determining a second link between a third data value stored in a third cell of the source dataset to a fourth data value stored in a fourth cell of the destination dataset of the stored tabular data comprises:
determining a difference between the third data value stored in the third cell of the source dataset and the fourth data value stored in a fourth cell of the destination dataset of the stored tabular data; and
in response to a determination that the difference is less than a difference threshold, determining that the second link is detected between the third data value in the third cell of the source dataset and the fourth data value in the fourth cell of the destination dataset, wherein the second link comprises a fuzzy match type of link.
18 . The method of claim 13 , further comprising determining a second link between a third data value stored in a third cell of the source dataset to a fourth data value stored in a fourth cell of the destination dataset of the stored tabular data comprises:
determining a first vector based at least in part on a first set of data values stored in cells in a third row of the source dataset, wherein the third row includes the third cell;
determining a second vector based at least in part on a second set of data values stored in cells in a fourth row of the destination dataset, wherein the fourth row includes the fourth cell;
comparing the first vector against the second vector to determine a similarity; and
in response to a determination that the similarity is greater than another threshold, determining that the second link is detected between the third row of the source dataset and the fourth row of the destination dataset, wherein the second link comprises a vector match type of link.