Machine learning based query optimization for federated databases
Techniques are provided for machine learning based query optimization for federated databases. An exemplary method comprises obtaining a query to be processed in a federated database; generating at least one predictive data movement instruction to move data to a target data source when the target data source satisfies one or more of a predefined efficiency criteria with respect to a query type of the query and a predefined capacity criteria at an expected execution time of the query; and generating a query execution plan for the query by calculating a cost of execution for a plurality of potential target data sources and selecting a target data source for the query based on the calculated cost of execution. The federated database optionally employs a dynamic federated query schema.
1. A method, comprising:
obtaining at least one query to be processed in a federated database;
generating at least one predictive data movement instruction to move data to a target data source when the target data source satisfies one or more of a predefined efficiency criterion with respect to a query type of said at least one query and a predefined capacity criteria at an expected execution time of said at least one query; and
generating, using at least one processing device, a query execution plan for said at least one query by calculating a cost of execution for a plurality of potential target data sources and selecting a target data source for said at least one query based on said calculated cost of execution, wherein the cost of execution comprises a predicted likelihood of the target data source being busy based on one or more of scheduler load information and historic load trends.
2. The method of claim 1 , further comprising the steps of linking a plurality of records by comparing records from two or more data sources to identify record pairs representing a substantially same real-world entity, and recording duplicate schema metadata as candidate matches.
3. The method of claim 1 , wherein said step of generating said at least one predictive data movement instruction further comprises predicting when said at least one query will be executed using scheduler batch load information.
4. The method of claim 1 , wherein said step of generating said query execution plan for said at least one query further comprises the step of evaluating whether said at least one query is a federated query.
5. The method of claim 1 , wherein said federated database employs a dynamic federated query schema.
6. The method of claim 1 , wherein the generating is performed in response to the target data source satisfying one or more of a predefined efficiency criterion with respect to a query type of said at least one query and a predefined capacity criterion at an expected execution time of said at least one query.
7. The method of claim 6 , wherein one or more of said predefined efficiency criterion for said target data source and said predefined capacity criteria for said target data source are evaluated using at least one machine learning classification.
8. A computer program product, comprising a non-transitory machine-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed by at least one processing device perform the following steps:
obtaining at least one query to be processed in a federated database;
generating at least one predictive data movement instruction to move data to a target data source when the target data source satisfies one or more of a predefined efficiency criterion with respect to a query type of said at least one query and a predefined capacity criteria at an expected execution time of said at least one query; and
generating, using at least one processing device, a query execution plan for said at least one query by calculating a cost of execution for a plurality of potential target data sources and selecting a target data source for said at least one query based on said calculated cost of execution, wherein the cost of execution comprises a predicted likelihood of the target data source being busy based on one or more of scheduler load information and historic load trends.
9. The computer program product of claim 8 , further comprising the steps of linking a plurality of records by comparing records from two or more data sources to identify record pairs representing a substantially same real-world entity, and recording duplicate schema metadata as candidate matches.
10. The computer program product of claim 8 , wherein said step of generating said at least one predictive data movement instruction further comprises predicting when said at least one query will be executed using scheduler batch load information.
11. The computer program product of claim 8 , wherein said step of generating said query execution plan for said at least one query further comprises the step of evaluating whether said at least one query is a federated query.
12. The computer program product of claim 8 , wherein said federated database employs a dynamic federated query schema.
13. The computer program product of claim 8 , wherein the generating is performed in response to the target data source satisfying one or more of a predefined efficiency criterion with respect to a query type of said at least one query and a predefined capacity criteria at an expected execution time of said at least one query.
14. An apparatus, comprising:
a memory; and
at least one processing device, coupled to the memory, operative to implement the following steps:
obtaining at least one query to be processed in a federated database;
generating at least one predictive data movement instruction to move data to a target data source when the target data source satisfies one or more of a predefined efficiency criterion with respect to a query type of said at least one query and a predefined capacity criteria at an expected execution time of said at least one query; and
generating, using at least one processing device, a query execution plan for said at least one query by calculating a cost of execution for a plurality of potential target data sources and selecting a target data source for said at least one query based on said calculated cost of execution, wherein the cost of execution comprises a predicted likelihood of the target data source being busy based on one or more of scheduler load information and historic load trends.
15. The apparatus of claim 14 , further comprising the steps of linking a plurality of records by comparing records from two or more data sources to identify record pairs representing a substantially same real-world entity, and recording duplicate schema metadata as candidate matches.
16. The apparatus of claim 14 , wherein said step of generating said at least one predictive data movement instruction further comprises predicting when said at least one query will be executed using scheduler batch load information.
17. The apparatus of claim 14 , wherein said step of generating said query execution plan for said at least one query further comprises the step of evaluating whether said at least one query is a federated query.
18. The apparatus of claim 14 , wherein said federated database employs a dynamic federated query schema.
19. The apparatus of claim 14 , wherein the generating is performed in response to the target data source satisfying one or more of a predefined efficiency criterion with respect to a query type of said at least one query and a predefined capacity criterion at an expected execution time of said at least one query.
20. The apparatus of claim 19 , wherein one or more of said predefined efficiency criterion for said target data source and said predefined capacity criteria for said target data source are evaluated using at least one machine learning classification.