Data ingestion to generate layered dataset interrelations to form a system of networked collaborative datasets
Various embodiments relate generally to data science and data analysis, and computer software and systems to provide an interface between repositories of disparate datasets and computing machine-based entities that seek access to the datasets, and, more specifically, to a computing and data storage platform that facilitates consolidation of one or more datasets, whereby data ingestion is performed to form data representing layered data files and data arrangements to facilitate, for example, interrelations among a system of networked collaborative datasets. In some examples, a method may include forming a first layer data file and a second layer data file, assigning addressable identifiers to uniquely identify units of data and data units to facilitate the linking of data, and implementing selectively one or more of a unit of data and a data unit as a function of a context of a data access request for a collaborative dataset.
1 . A method comprising:
receiving data into a dataset ingestion controller configured to form a collaborative dataset, the data representing a set of data having an initial data format;
characterizing the set of data to classify a plurality of subsets of data including same or equivalent data;
compressing the plurality of subsets of data to form compressed representations of the plurality of subsets of data;
filtering probabilistically a particular subset of data in the plurality of subsets of data to determine the particular subset of data is in the set of data;
adapting the initial data format of the set of data to form a dataset having a first data format;
forming a multiple layer data file to link other data in a second data format to the dataset in the first data format;
analyzing the particular subset of the data to determine a dataset attribute, wherein a machine learning algorithm is configured to automatically predict a data value as the dataset attribute;
generating descriptor data based on the dataset attribute associated with the particular subset of the data;
converting the dataset from the first data format using a format converter configured to form an atomized dataset in a first graph data arrangement, the atomized dataset comprising the collaborative dataset;
associating a unit of the descriptor data with a unit of supra-descriptor data to form an association, the supra-descriptor data being enhanced to include one or more additional units of descriptor data derived from the set of data having the initial data format, the one or more additional units of descriptor data comprising one or more additional dataset attributes of the set of data having the initial data format; and
forming a second graph data arrangement including the supra-descriptor data and the association with the descriptor data.
2 . The method of claim 1 , wherein:
forming the multiple layer data file comprises forming a first layer data file to reference the other data in the second data format in which a unit of data in the other data in the second data format is configured to link with another data file; and
forming a second layer data file that includes another particular subset of data based on the other data in the second data format, in which a unit of data in the other particular subset of data in the second data format being configured to link to a unit of data in the first data format.
3 . The method of claim 1 , further comprising:
assigning an addressable identifier to uniquely identify a unit of data to facilitate linking data between the dataset in the first data format and the other data in the second data format, wherein at least one of the addressable identifiers references a triplestore database; and
selectively implementing the unit of data as a function of context of a data access request.
4 . The method of claim 1 , wherein analyzing the particular subset of the data to determine the dataset attribute comprises determining ancillary data regarding the particular subset of the data.
5 . The method of claim 1 , wherein the atomized dataset comprises atomized descriptor data.
6 . The method of claim 1 , wherein the atomized dataset comprises atomized source data.
7 . The method of claim 1 , wherein the supra-descriptor data includes additional units of descriptor data derived from the set of data.
8 . The method of claim 1 wherein compressing the plurality of subsets of data to form the compressed representations of the plurality of subsets of data comprises applying a minhashing algorithm to determine a degree of similarity between the compressed representations of the plurality of subsets of data.
9 . The method of claim 1 wherein compressing the plurality of subsets of data to form the compressed representations of the plurality of subsets of data comprises identifying a categorical variable.
10 . The method of claim 9 wherein the categorical variable is associated with a column of data including the particular subset of data.
11 . The method of claim 1 wherein filtering probabilistically the particular subset of data in the plurality of subsets of data to determine the particular subset of data is in the set of data comprises implementing one or more Bloom filters.
12 . A system comprising:
a memory including executable instructions; and
a processor responsive to executing instructions, is configured to:
receive data into a dataset ingestion controller configured to form a collaborative dataset, the data representing a set of data having an initial data format;
characterize the set of data to classify a plurality of subsets of data including same or equivalent data;
compress the plurality of subsets of data to form compressed representations of the plurality of subsets of data;
filter probabilistically a particular subset of data in the plurality of subsets of data to determine the particular subset of data is in the set of data;
adapt the initial data format of the set of data to form a dataset having a first data format;
form a multiple layer data file to link other data in a second data format to the dataset in the first data format;
analyze the particular subset of the data to determine a dataset attribute, wherein a machine learning algorithm is configured to automatically predict a data value as the dataset attribute;
generate descriptor data based on the dataset attribute associated with the particular subset of the data;
convert the dataset from the first data format using a format converter configured to form an atomized dataset in a first graph data arrangement, the atomized dataset comprising the collaborative dataset;
associate a unit of the descriptor data with a unit of supra-descriptor data to form an association, the supra-descriptor data being enhanced to include one or more additional units of descriptor data derived from the set of data having the initial data format, the one or more additional units of descriptor data comprising one or more additional dataset attributes of the set of data having the initial data format; and
form a second graph data arrangement including the supra-descriptor data and the association with the descriptor data.
13 . The system of claim 12 , wherein the processor is further configured to:
form a first layer data file to reference the other data in the second data format in which a unit of data in the other data in the second data format is configured to link with another data file; and
form a second layer data file that includes another particular subset of data based on the other data in the second data format, in which a unit of data in the other particular subset of data in the second data format being configured to link to a unit of data in the first data format.
14 . The system of claim 12 , wherein the processor is further configured to:
assign an addressable identifier to uniquely identify a unit of data to facilitate linking data between the dataset in the first data format and the other data in the second data format, wherein at least one of the addressable identifiers references a triplestore database; and
selectively implement the unit of data as a function of context of a data access request.
15 . The system of claim 12 , wherein the processor is further configured to determine ancillary data regarding the particular subset of the data.
16 . The system of claim 12 , wherein the atomized dataset comprises atomized descriptor data.
17 . The system of claim 12 , wherein the atomized dataset comprises atomized source data.
18 . The system of claim 12 , wherein the supra-descriptor data includes additional units of descriptor data derived from the set of data.
19 . A non-transitory computer readable medium having one or more computer program instructions configured to perform a method, the method comprising:
receiving data into a dataset ingestion controller configured to form a collaborative dataset, the data representing a set of data having an initial data format;
characterizing the set of data to classify a plurality of subsets of data including same or equivalent data;
compressing the plurality of subsets of data to form compressed representations of the plurality of subsets of data;
filtering probabilistically a particular subset of data in the plurality of subsets of data to determine the particular subset of data is in the set of data;
adapting the initial data format of the set of data to form a dataset having a first data format;
forming a multiple layer data file to link other data in a second data format to the dataset in the first data format;
analyzing the particular subset of the data to determine a dataset attribute, wherein a machine learning algorithm is configured to automatically predict a data value as the dataset attribute;
generating descriptor data based on the dataset attribute associated with the particular subset of the data;
converting the dataset from the first data format using a format converter configured to form an atomized dataset in a first graph data arrangement, the atomized dataset comprising the collaborative dataset;
associating a unit of the descriptor data with a unit of supra-descriptor data to form an association, the supra-descriptor data being enhanced to include one or more additional units of descriptor data derived from the set of data having the initial data format, the one or more additional units of descriptor data comprising one or more additional dataset attributes of the set of data having the initial data format; and
forming a second graph data arrangement including the supra-descriptor data and the association with the descriptor data.