Determining a degree of similarity of a subset of tabular data arrangements to subsets of graph data arrangements at ingestion into a data-driven collaborative dataset platform
Various techniques are described, including evaluating ingested data including a dataset to identify one or more links to other datasets stored in a graph, using a similarity determination algorithm to identify a degree of similarity between datasets to determine joinability of ingested datasets with graph-stored datasets, determining a ratio to determine whether to perform an overlap or coverage function, associating a subset of similarity matrices with a subset of graph data joined to the ingested dataset, and forming links in a column of data between the dataset and the another dataset of the ingested data based on the degree of similarity.
1. A method, comprising:
ingesting a dataset to form an ingested dataset;
compressing the ingested dataset in accordance with at least one of one or more algorithmic hash functions to form a compressed data representation of the ingested data, the at least one of the one or more algorithmic hash functions including a plurality of differently-configured hash functions to generate a first subset of instances in a first state relative to a second subset of instances in a second state to specify a degree of similarity, the plurality of the differently-configured hash functions being configured to generate a plurality of hash values, at least one of the plurality of differently-configured hash functions being implemented based on a data type;
identifying an indication associated with instructing computation of the degree of similarity between the ingested dataset and another dataset, the degree of similarity being used to join the dataset and the another dataset in a graph in response to executing instructions at one or more processors;
determining a first ratio associated with an overlap function;
determining a second ratio associated with a coverage function;
identifying a metric configured to be used to determine the degree of similarity;
associating a subset of similarity matrices with a subset of graph data joined to the ingested dataset;
accessing the subset of similarity matrices, at least one subset of a similarity matrix is formed to identify a subset of relevant data associated with the another dataset disposed in a graph data arrangement, at least a portion of the another dataset in the graph data arrangement being formatted as one or more triple-based data formats, the similarity matrix and the degree of similarity being a function of the plurality of the hash values relating to a union of compressed ingested dataset data and compressed target data;
forming a plurality of links among a column of data associated with the ingested dataset as the dataset and the another dataset of the ingested data, the plurality of links being determined based on the degree of similarity based on the compressed data representation; and
receiving a query as a data operation based on ranked datasets identified based on the degree of similarity and other degrees of similarity,
wherein a combined number of hash-derived attributes of the at least one of one or more algorithmic hash functions are configured to determine the degree of similarity based on multiple algorithmic hash functions.
2. The method of claim 1 , wherein the determining the second ratio is determined if the overlap function is not performed.
3. The method of claim 1 , wherein the determining the first ratio indicates the coverage function is not performed.
4. The method of claim 1 , wherein the metric comprises a threshold degree of similarity configured to distinguish the dataset from the another dataset to identify a degree of joinability.
5. The method of claim 1 , wherein the metric comprises a threshold degree of similarity configured to generate a ranking including the dataset and the another dataset.
6. The method of claim 1 , wherein the metric comprises a threshold degree of similarity configured to generate a ranking including the dataset and the another dataset, the ranking being used to identify one or more degree of joinability between the dataset and the another dataset.
7. The method of claim 1 , wherein the metric comprises a threshold degree of similarity configured to generate a ranking including the dataset and the another dataset, the ranking being used to identify one or more degree of joinability between the dataset and the another dataset to identify one or more of the plurality of links.
8. The method of claim 1 , wherein the dataset is configured to be presented in a display if the degree of similarity exceeds a threshold.
9. The method of claim 1 , wherein the another dataset is configured to be presented in a display if the degree of similarity exceeds a threshold.
10. The method of claim 1 , further comprising analyzing the dataset and the another dataset to determine joinability if the degree of similarity is compliant.
11. The method of claim 1 , wherein the degree of similarity is compliant and the dataset and the another dataset are identified as joinable.
12. The method of claim 1 , wherein the degree of similarity is configured to identify a priority associated with the dataset and another priority associated with the another dataset, the priority and the another priority being used to form the plurality of links.
13. A method, comprising:
ingesting a dataset to form an ingested dataset;
compressing the ingested dataset in accordance with at least one of one or more algorithmic hash functions to form a compressed data representation of the ingested data, the at least one of the one or more algorithmic hash functions including a plurality of differently-configured hash functions to generate a first subset of instances in a first state relative to a second subset of instances in a second state to specify a degree of similarity, the plurality of hash functions being configured to generate a plurality of hash values, at least one of the plurality of differently-configured hash functions being implemented based on a data type;
evaluating the ingested data including a dataset and another dataset to identify one or more links to other datasets stored in a graph;
using a similarity determination algorithm to identify the degree of similarity between the dataset, the another dataset, and the other datasets to determine joinability of the dataset, the another dataset, and the other datasets in the graph, the similarity determination algorithm including executable instructions processed at one or more processors;
determining a ratio, the ratio being used to determine whether to perform a function;
associating a subset of similarity matrices with a subset of graph data joined to the ingested dataset;
accessing the subset of similarity matrices, at least one subset of a similarity matrix is formed to identify a subset of relevant data associated with the another dataset disposed in a graph data arrangement, at least a portion of the another dataset in the graph data arrangement being formatted as one or more triple-based data formats, the similarity matrix and the degree of similarity being a function of the plurality of the hash values relating to a union of compressed ingested dataset data and compressed target data;
forming a plurality of links among a column of data associated with the ingested dataset as the dataset and the another dataset of the ingested data, the plurality of links being determined based on the degree of similarity based on the compressed data representation; and
receiving a query as a data operation based on ranked datasets identified based on the degree of similarity and other degrees of similarity,
wherein a combined number of hash-derived attributes of the at least one of one or more algorithmic hash functions are configured to determine the degree of similarity based on multiple algorithmic hash functions.
14. The method of claim 13 , wherein the similarity determination algorithm is configured to use a metric to determine the degree of similarity.
15. The method of claim 13 , wherein the function is an overlap function.
16. The method of claim 13 , wherein the function is a coverage function.
17. A system, comprising:
a data repository including memory configured to store data associated with a graph stored in a data arrangement with which at least a portion is configured to store graph-based data arrangements; and
a collaborative dataset consolidation system including one or more processors configured to ingest a dataset to form an ingested dataset, compress the ingested dataset in accordance with at least one of one or more algorithmic hash functions to form a compressed data representation of the ingested data, the at least one of the one or more algorithmic hash functions including a plurality of differently-configured hash functions to generate a first subset of instances in a first state relative to a second subset of instances in a second state to specify a degree of similarity, the plurality of hash functions being configured to generate a plurality of hash values, at least one of the plurality of differently-configured hash functions being implemented based on a data type, evaluate the ingested data including a dataset and another dataset to identify one or more links to other datasets stored in a graph, to use a similarity determination algorithm to identify the degree of similarity between the dataset, the another dataset, and the other datasets to determine joinability of the dataset, the another dataset, and the other datasets in the graph, to determine a ratio, the ratio being used to determine whether to perform a function, to associate a subset of similarity matrices with a subset of graph data joined to the ingested dataset, access the subset of similarity matrices, at least one subset of a similarity matrix is formed to identify a subset of relevant data associated with the another dataset disposed in a graph data arrangement, at least a portion of the another dataset in the graph data arrangement being formatted as one or more triple-based data formats, the similarity matrix and the degree of similarity being a function of the plurality of the hash values relating to a union of compressed ingested dataset data and compressed target data; to form a plurality of links among a column of data between the ingested dataset and the another dataset, the plurality of links being determined based on the degree of similarity based on the compressed data representation, and to receive a query as a data operation based on ranked datasets identified based on the degree of similarity and other degrees of similarity,
wherein a combined number of hash-derived attributes of the at least one of one or more algorithmic hash functions are configured to determine the degree of similarity based on multiple algorithmic hash functions.
18. The system of claim 17 , wherein a dataset ingestion controller is configured to evaluate the ingested data including the dataset and the another dataset.
19. The system of claim 17 , wherein the plurality of links are formed between the dataset and the another dataset generating referential data in a graph data arrangement.
20. The system of claim 17 , wherein the collaborative dataset consolidation system is configured to generate referential data stored in a graph data arrangement comprising a datum, the datum being configured to identify a degree of similarity between the dataset and the another dataset based on the degree of similarity determined by the similarity determination algorithm.