Loading collaborative datasets into data stores for queries via distributed computer networks
Various embodiments relate generally to data science and data analysis, computer software and systems, and wired and wireless network communications to provide an interface between repositories of disparate datasets and computing machine-based entities that seek access to the datasets, and, more specifically, to a computing and data storage platform that facilitates consolidation of one or more datasets, whereby a collaborative data layer and associated logic facilitate, for example, efficient access to, and implementation of, collaborative datasets. In some examples, a system may include an atomized workflow loader configured to receive an atomized dataset to load into a data store, and to determine resource requirements data to describe at least one resource requirement. The atomized workflow loader may be further configured to select a data store type based on a resource requirement, and perform a load operation of the atomized dataset as a function of the data store type.
1. A method comprising:
receiving an atomized dataset to load into a graph-based data store, the atomized dataset including a data arrangement in which data is stored as an atomized data point with one or more other atomized data points of one or more data types as a consolidated dataset, the atomized data point being implemented as a triple, the data arrangement representing at least a portion of a graph, the atomized data point being a representation for a relationship between two data units, and the consolidated dataset having a plurality of atomized data points of the one or more data types also having links that, when parsed, identify one or more relationships between the plurality of atomized data points and the one or more data types including a resource associated with each of the atomized and the other atomized data points and a data type associated with the resource;
converting the atomized dataset, after being received, from a first data format to a second data format, the second data format being a collaborative data format configured to be used to form a portion of the graph;
determining resource requirements data to describe a capability to operate a database configured to access graph-based data to identify at least one resource requirement;
selecting a data store type based on the at least one resource requirement;
performing a load operation of the atomized dataset as a function of the data store;
managing versioning of the atomized dataset to include a hierarchy of files, comprising tracking each version as one of an immutable collection of data files;
receiving a query to access the atomized dataset;
classifying at least a portion of the query directed to the dataset to determine a classification type, whereby the classification type is associated with a type of query for a query portion associated with a specific entity; and
applying the portion of the query as a sub-query to at least one of a number of data stores, a subset of which includes the one or more types of triplestore-based graph databases.
2. The method of claim 1 wherein tracking each version comprises:
tracking at least one pointer to at least one of the data files.
3. The method of claim 1 wherein performing the load operation of the atomized dataset comprises:
loading the dataset into a graph database.
4. The method of claim 1 wherein determining the resource requirements comprises:
identifying operating characteristics of a data store to load the atomized dataset as graph data.
5. The method of claim 4 wherein determining the resource requirements comprises:
determining an operating characteristic of the data store related to a text search; and
identifying the data store for selection as a function of the classification type.
6. The method of claim 4 wherein determining the resource requirements comprises:
determining an operating characteristic of the data store related to geo-spatial information; and
identifying the data store for selection as a function of the classification type.
7. The method of claim 4 wherein determining the resource requirements comprises:
determining an operating characteristic of the data store related to graphic processing unit (“GPU”)-optimized data; and
identifying the data store for selection as a function of the classification type.
8. The method of claim 1 wherein selecting the data store type comprises:
selecting a product having a proprietary storage architecture.
9. The method of claim 8 wherein selecting the product comprises:
selecting a triple having a specific storage architecture.
10. A system comprising:
a processor and a memory to store one or more executable instructions, the processor configured to execute instructions to implement an atomized workflow loader configured to receive an atomized dataset to load into a graph-based data store, the atomized dataset including a data arrangement in which data is stored as an atomized data point with one or more other atomized data points of one or more data types as a consolidated dataset, the atomized data point being implemented as a triple, the data arrangement representing at least a portion of a graph, the atomized data point being a representation for a relationship between two data units, and the consolidated dataset having a plurality of atomized data points of the one or more data types also having links that, when parsed, identify one or more relationships between the plurality of atomized data points and the one or more data types including a resource associated with each of the atomized and the other atomized data points and a data type associated with the resource, to convert the atomized dataset, after being received, from a first data format to a second data format, the second data format being a collaborative data format configured to be used to form a portion of the graph, to determine resource requirements data to describe a capability to operate a database configured to access graph-based data to identify at least one resource requirement, the atomized workflow loader further configured to select a data store type based on the at least one resource requirement, perform a load operation of the atomized dataset as a function of the data store type, manage versioning of the atomized dataset to include a hierarchy of files, comprising a version controller configured to track each version as one of an immutable collection of data files to manage versioning of the atomized dataset, receive a query to access the atomized dataset, classify at least a portion of the query directed to the dataset to determine a classification type, whereby the classification type is associated with a type of query for a query portion associated with a specific entity, and apply the portion of the query as a sub-query to at least one of a number of data stores, a subset of which includes the one or more types of triplestore-based graph databases.
11. The system of claim 10 where in the version controller is further configured to:
track at least one pointer to at least one of the data files.
12. The system of claim 10 further comprising:
a dataset requirement determinator configured to identify operating characteristics of a data store to load the atomized dataset as graph data.
13. The system of claim 12 wherein the data store is a triple store.
14. The system of claim 12 wherein the dataset requirement determinator is configured to:
determine an operating characteristic of the data store related to a text search; and
identify the data store for selection.
15. The system of claim 12 wherein the dataset requirement determinator is configured to:
determine an operating characteristic of the data store related to geo-spatial information; and
identify the data store for selection.
16. The system of claim 12 wherein the dataset requirement determinator is configured to:
determine an operating characteristic of the data store related to graphic processing unit (“GPU”)-optimized data; and
identify the data store for selection.
17. The system of claim 10 further comprises:
a product selector configured to select a product having a proprietary storage architecture, the product selector is further configured to select a triple having a specific storage architecture.
18. The system of claim 10 further comprising:
a data representation including one or more models configured to direct how to load datasets into the graph-based data store.