Symbiotic data insights from harmonized queries
Aspects of the present disclosure relate to systems and methods for multi-data source query implementation and aggregation. In embodiments, the query aggregation system comprises a data abstraction engine, a query controller or scheduler and an aggregator. In embodiments, the data abstraction engine normalizes data schemas from multiple data sources based on the parameter requirements. In embodiments, the query controller schedules and runs normalized queries against each data sources independently and securely. In embodiments, the aggregator collects and merges the aggregates of the results into a single analysis. In embodiments, the query aggregation system communicates to various data sources including enterprise data silo(s) and personal data silo(s) to send queries and receive query results. In embodiments, the query sent to each data source may be specialized or tailored for the specific requirement of the data source.
1. A computer-implemented method for multi-data source query implementation and aggregation comprising:
receiving a set of one or more search parameters from a user interface;
normalizing data schemas from multiple data sources based on the one or more search parameters;
distributing individual queries toward each data source;
running each individual query in respective data source to obtain individual query result;
collecting each individual query result and reassembling collected query results into a single analysis while maintaining any access restriction of each query result within the single analysis such that a user may only access data associated with the user;
creating a query package that includes each individual query; and
storing the query package in a data storage with an associated unique identifier, wherein the query package includes one or more visualizations of queried data.
2. The computer-implemented method of claim 1 wherein the multiple data sources comprising at least one enterprise data sources and at least one personal data sources.
3. The computer-implemented method of claim 1 wherein the step of normalizing data schemas from multiple data sources comprising normalizing a query template for data format matching.
4. The computer-implemented method of claim 3 wherein the step of normalizing data schemas from multiple data sources comprising automatically and manually matching the query template.
5. The computer-implemented method of claim 1 wherein collecting each individual query result and reassembling collected query results into a single analysis comprises verifying completeness of collected query results.
6. The computer-implemented method of claim 5 wherein the set of reassembling collected query results into a single analysis is implemented after the completeness of collected query results is verified or after a predetermined time interval.
7. The computer-implemented method of claim 6 , wherein the access restriction is maintained within the single analysis.
8. The computer-implemented method of claim 1 wherein running each individual query in respective data source is operated via each corresponding aggregate query agent.
9. The computer-implemented method of claim 8 wherein the aggregate query agent comprising a source query manager to apply access limitations of the query and an access controller to facilitate access to corresponding data source.
10. A system for multi-data source query implementation and aggregation comprising:
a memory; and
one or more processors configured to execute a set of instructions comprising:
normalize a query template for multiple data sources based on the one or more query parameters input via a user interface;
distribute individual queries toward the multiple data sources, each data source comprising an aggregate query agent to receive one individual query, implement the individual query and obtain one individual query result;
collect the individual query result from each data source and reassembling collected query results into a single analysis while maintaining any access restriction of each query result within the single analysis such that a user may only access data associated with the user;
create query package that includes each individual query; and
cause the query package to be stored in a data storage with an associated unique identifier, wherein the query package includes one or more visualizations of queried data.
11. The system of claim 10 wherein the multiple data sources comprising at least one enterprise data source and at least one personal data source.
12. The system of claim 11 wherein the enterprise data sources comprising a plurality of different types of data sources.
13. The system of claim 10 wherein the query template is normalized automatically or manually for data format matching.
14. The system of claim 10 wherein the individual queries toward each data source are scheduled and distributed according to an arrangement based at least on size or data traffic of each data source.
15. The system of claim 10 wherein the individual query results are collected according to a schedule determined based on at least a parameter selected from the size the query result, the data traffic of each date source, a time interval pre-assigned to each aggregate query agent for the individual query.
16. A computer-implemented method comprising:
receiving an individual query to a database within a data silo with one or more search parameters;
regulating access to the database or a subset of the database within the within a data silo based on the one or more search parameters;
facilitating access to corresponding the databases or a subset of the database to generate a query result;
providing the query result to an aggregator for query aggregation as a part of a single analysis while maintaining any access restriction of each query result within the single analysis such that a user may only access data associated with the user;
creating a query package that includes the query result; and
causing the query package to be stored in a data storage with an associated unique identifier, wherein the query package includes one or more visualizations of queried data.
17. The computer-implemented method of claim 16 wherein the individual query is specialized for accessing requirement of the database.
18. The computer-implemented method of claim 16 wherein the access restriction is maintained within the single analysis.
19. The computer-implemented method of claim 16 wherein the query result is provided to an aggregator according to a receiving schedule of the aggregator.
20. The computer-implemented method of claim 19 wherein the receiving schedule is determined based on at least one parameter selected from the size the query result, the data traffic of the data silo, a time interval pre-assigned for the individual query.