Collaborative dataset consolidation via distributed computer networks
Various embodiments relate generally to data science and data analysis, computer software and systems, and wired and wireless network communications to provide an interface between repositories of disparate datasets and computing machine-based entities that seek access to the datasets, and, more specifically, to a computing and data storage platform that facilitates consolidation of one or more datasets, whereby a collaborative data layer and associated logic facilitate, for example, efficient access to, and implementation of, collaborative datasets. In some examples, a method may include receiving data representing a query into a collaborative dataset consolidation system, identifying datasets relevant to the query, generating one or more queries to access disparate data repositories, and retrieving data representing query results. In some cases, one or more queries are applied (e.g., as a federated query) to atomized datasets stored in one or more atomized data stores, at least two of which may be different.
1. A method comprising:
receiving a data file including a dataset into a collaborative dataset consolidation system responsive to receiving a user input signal originating a first user interface;
formatting the dataset to form a first atomized dataset including atomized data points each including data representing at least two objects and an association between the two objects;
forming a second atomized dataset including the first atomized dataset and one or more other atomized datasets;
receiving data representing a federated query into the collaborative dataset consolidation system, the query being associated with an identifier;
identifying a subset of the second atomized dataset relevant to the federated query, wherein portions of the second atomized dataset are disposed in different data repositories;
generating results of the federated query by generating one or more sub-queries configured to identify, based on the association, at least a portion of the first atomized dataset and at least another portion of the one or more other atomized datasets, and to retrieve the at least the portion of the first atomized dataset and the at least the another portion of the one or more other atomized datasets, the at least the portion of the first atomized dataset and the at least the another portion of the one or more other atomized datasets being transformed to a format and retrieved from the different data repositories; and
generating data representing a notification that the second atomized dataset is queried.
2. The method of claim 1 wherein further comprising:
generating a plurality of sub-queries each of which is configured to access at least one of the different data repositories.
3. The method of claim 2 wherein generating the plurality of sub-queries further comprises:
receiving data representing the federated query.
4. The method of claim 1 wherein further comprising:
retrieving data representing query results from the at least one of the different data repositories.
5. The method of claim 1 further comprising transmitted the data representing the notification to a second user interface.
6. The method of claim 1 further comprising transmitted the data representing the notification to an activity feed portion of a second user interface.
7. The method of claim 1 wherein generating the plurality of sub-queries comprises: classifying query portions.
8. The method of claim 7 wherein classifying the query portions comprises:
identifying a classification type for a portion of the query.
9. The method of claim 1 wherein the datasets comprise linked data points.
10. The method of claim 9 wherein linked data points comprise triples.
11. The method of claim 10 wherein at least one triple of the triples are formatted to comply with a Resource Description Framework (“RDF”) data model.
12. A system comprising:
a data store configured to store executable instructions and data; and
a processor configured to execute instructions, the processor being configured to:
receive a data file including a dataset into a collaborative dataset consolidation system responsive to receiving a user input signal originating a first user interface;
format the dataset to form a first atomized dataset including atomized data points each including data representing at least two objects and an association between the two objects;
form a second atomized dataset including the first atomized dataset and one or more other atomized datasets;
receive data representing a query into the collaborative dataset consolidation system, the query being associated with an identifier;
identify a subset of the second atomized dataset relevant to the query, wherein portions of the second atomized dataset are disposed in different data repositories;
generate a plurality of sub-queries each of which is configured to access at least one of the different data repositories, by generating the plurality of sub-queries, each of the plurality of sub-queries being configured to identify, based on the association, at least a portion of the first atomized dataset and at least another portion of the one or more other atomized datasets, and to retrieve the at least the portion of the first atomized dataset and the at least the another portion of the one or more others atomized dataset, the at least the portion of the first atomized dataset and the at least the another portion of the one or more other atomized datasets being transformed to a format and retrieved from the different data repositories; and
retrieving data representing query results from the at least one of the different data repositories.
13. The system of claim 12 wherein the processor is further configured to:
generate the query results responsive to a federated query; and
generate data representing a notification that the second atomized dataset is queried.
14. The system of claim 12 wherein the processor is further configured to:
transmit the data representing the notification to a second user interface.
15. The system of claim 12 wherein the processor is further configured to:
transmit the data representing the notification to an activity feed portion of a second user interface.
16. The system of claim 12 wherein the processor is further configured to:
generate the plurality of sub-queries comprises: classifying query portions.
17. The system of claim 16 wherein the processor is further configured to:
classify the query portions including identifying a classification type for a portion of the query.
18. The system of claim 12 wherein the datasets comprise linked data points.
19. The system of claim 18 wherein linked data points comprise triples.
20. The system of claim 19 wherein at least one triple of the triples are formatted to comply with a Resource Description Framework (“RDF”) data model.