Dynamically optimizing data access patterns using predictive crowdsourcing
A system, method, and computer-readable medium for performing a dynamic data index restructure operation. The dynamic data index restructure operation restructures the data indexes within a business intelligence architecture based on usage patterns so as to support queries most likely to be generated by users of the business intelligence architecture. In certain embodiments, the dynamic data index restructure operation is performed via a dynamic data index restructure agent which may be positioned between a business intelligence application and the business intelligence database.
1. A computer-implementable method for performing a dynamic data index restructure operation within a business intelligence architecture, comprising:
determining usage patterns of queries within the business intelligence architecture, wherein the usage patterns are determined using predictive crowdsourcing to estimate the usage patterns of queries;
identifying a data index used for generating a query within the business intelligence architecture; and
restructuring the data index within the business intelligence architecture based on the estimated usage patterns and based on predefined criteria, the restructuring supporting queries that are predicted to be generated, but not yet generated, by users of the business intelligence architecture, the restructuring comprising pre-processing at least a portion of data included in the data index corresponding to queries that are predicted to be generated, but not yet generated, by users of the business intelligence architecture, the at least the portion of data included in the data index being restructured based on predefined criteria, the restructuring including identifying a wide data file and reshaping the data file into long form.
2. The method of claim 1 , wherein:
the determining usage patterns is performed via a dynamic data index restructure agent.
3. The method of claim 2 , wherein:
the dynamic data index restructure agent is positioned between a business intelligence application and a business intelligence database.
4. The method of claim 1 , wherein:
the determining usage patterns is performed via a predictive engine, the predictive engine speculating data access patterns based on a certain number of retrieved queries from users accessing a database within the business intelligence architecture.
5. The method of claim 4 , wherein:
the predictive engine is customizable to retrieve query patterns based upon predefined criteria.
6. The method of claim 1 , wherein the pre-processing at least the portion of data included in the data index corresponding to queries that are predicted to be generated, comprises at least one of:
reshaping the portion of data included in the data index corresponding to queries that are predicted to be generated included in the data index into long form;
stacking the portion of data included in the data index corresponding to queries that are predicted to be generated;
sorting the portion of data included in the data index corresponding to queries that are predicted to be generated; and
unstacking the portion of data included in the data index corresponding to queries that are predicted to be generated.
7. A system, comprising:
a processor;
a data bus coupled to the processor; and
a non-transitory, computer-readable storage medium embodying computer program code, the non-transitory, computer-readable storage medium being coupled to the data bus, the computer program code interacting with a plurality of computer operations and comprising instructions executable by the processor and configured for:
determining usage patterns of queries within a business intelligence architecture;
identifying a data index used for generating a query within the business intelligence architecture, wherein the usage patterns are determined using predictive crowdsourcing to estimate the usage patterns of queries; and
restructuring the data index within the business intelligence architecture based on the estimated usage patterns and based on predefined criteria, the restructuring supporting queries that are predicted to be generated, but not yet generated, by users of the business intelligence architecture, the restructuring comprising pre-processing at least a portion of data included in the data index corresponding to queries that are predicted to be generated, but not yet generated, by users of the business intelligence architecture, the at least the portion of data included in the data index being restructured based on predefined criteria, the restructuring including identifying a wide data file and reshaping the data file into long form.
8. The system of claim 7 , wherein:
the determining usage patterns is performed via a dynamic data index restructure agent.
9. The system of claim 8 , wherein:
the dynamic data index restructure agent is positioned between a business intelligence application and a business intelligence database.
10. The system of claim 7 , wherein;
the determining usage patterns is performed via a predictive engine, the predictive engine speculating data access patterns based on a certain number of retrieved queries from users accessing a database within the business intelligence architecture.
11. The system of claim 10 , wherein:
the predictive engine is customizable to retrieve query patterns based upon predefined criteria.
12. The system of claim 7 , wherein the pre-processing at least the portion of data included in the data index corresponding to queries that are predicted to be generated, comprises at least one of:
reshaping the portion of data included in the data index corresponding to queries that are predicted to be generated included in the data index into long form;
stacking the portion of data included in the data index corresponding to queries that are predicted to be generated;
sorting the portion of data included in the data index corresponding to queries that are predicted to be generated; and
unstacking the portion of data included in the data index corresponding to queries that are predicted to be generated.
13. A non-transitory, computer-readable storage medium embodying computer program code, the computer program code comprising computer executable instructions configured for:
determining usage patterns of queries within a business intelligence architecture, wherein the usage patterns are determined using predictive crowdsourcing to estimate the usage patterns of queries;
identifying a data index used for generating a query within the business intelligence architecture; and
restructuring the data index within the business intelligence architecture based on the estimated usage patterns and based on predefined criteria, the restructuring supporting queries that are predicted to be generated, but not yet generated, by-users of the business intelligence architecture, the restructuring comprising pre-processing at least a portion of data included in the data index corresponding to queries that are predicted to be generated, but not yet generated, by users of the business intelligence architecture, the at least the portion of data included in the data index being restructured based on predefined criteria, the restructuring including identifying a wide data file and reshaping the data file into long form.
14. The non-transitory, computer-readable storage medium of claim 13 , wherein:
the determining usage patterns is performed via a dynamic data index restructure agent.
15. The non-transitory, computer-readable storage medium of claim 14 , wherein:
the dynamic data index restructure agent is positioned between a business intelligence application and a business intelligence database.
16. The non-transitory, computer-readable storage medium of claim 13 , wherein:
the determining usage patterns is performed via a predictive engine, the predictive engine speculating data access patterns based on a certain number of retrieved queries from users accessing a database within the business intelligence architecture.
17. The non-transitory, computer-readable storage medium of claim 16 , wherein:
the predictive engine is customizable to retrieve query patterns based upon predefined criteria.