Method and system for providing a context-sensitive, non-intrusive data processing optimization framework
A method of performing a data search in a data source by which an operator of a data search pipeline is just-in-time optimized and compiled, using an operator optimization module which optimizes and compiles an intermediate representation of the operator, considering runtime information, and optimization rules, to produce an operator that is optimized for the data search being performed. The method can be applied with one operator or with many operators applied in any sequence or tree structure according to a query plan, as determined by runtime information and optimization rules.
1 . A computerized method of performing, by a query processing engine, a data search in a database:
obtaining, by an operator specialization module of the query processing engine, intermediate representation code of at least one operator, the intermediate representation code configured in a high-level programming language and compiled by a front-end compiler;
optimizing, by the operator specialization module and during runtime of performing the data search, the intermediate representation code based on runtime information of the runtime of performing the data search, to generate at least one optimized and compiled operator, the information of the runtime of performing the data search comprising statistical information about the data search of the data related thereto, the optimizing the intermediate representation code comprising:
specializing, based on the information of the runtime of performing the data search, the intermediate representation code for adapting to the data search and the hardware of the query processing engine,
selecting a level of parallelism and an operator fusion mechanism using the statistical information, and
optimizing the specialized intermediate representation code based on the information of the runtime of performing the data search to generate at least one optimized and compiled operator,
compiling the optimized intermediate representation code to generate at least one optimized and compiled operator; and
performing the data search by the query processing engine using the at least one optimized and compiled operator.
2 . The method of claim 1 , wherein the at least one operator comprises annotations tagging functions that are to be specialized when the at least one operator is being optimized and compiled for the data search at the runtime of performing the data search.
3 . The method of claim 1 , wherein the runtime information of the runtime of performing the data search further comprises:
data layout of the data search or data related thereto,
statistical information about the data search or data related thereto, or
a combination thereof.
4 . The method of claim 1 , wherein the optimization rules include global algorithms.
5 . The method of claim 1 , wherein the optimization rules are based on the runtime information of the runtime of performing the data search.
6 . The method of claim 1 , wherein the optimization rules include algorithms specific to the at least one operator.
7 . The method of claim 1 , wherein optimizing the at least one operator comprises an application of at least one optimization to a code of the at least one operator that increases the efficiency of a data search executing the at least one operator.
8 . The method of claim 1 , wherein the performing the data search comprises generating optimized and compiled operators via the optimizing and compiling, which are subsequently executed by the data search according to a query plan defined by the data search.
9 . A system for performing, by a query processing engine of specific hardware, a data search in a database comprising:
a memory storing instructions; and
at least one processor in communication with the memory, the at least one processor configured, upon execution of the instructions, to perform the following steps:
obtaining intermediate representation code of at least one operator, the intermediate representation code configured in a high-level programming language and compiled by a front-end compiler;
optimizing the intermediate representation code, based on information of the runtime of performing the data search;
compiling the optimized intermediate representation code to generate at least one optimized and compiled operator, and
performing the data search by the query processing engine using the at least one optimized and compiled operator,
wherein the information of the runtime of performing the data search comprises statistical information about the data search or the data related thereto; and
wherein the optimizing the intermediate representation code comprises:
specializing, based on the information of the runtime of performing the data search, the intermediate representation code for adapting to the data search and the hardware of the query processing engine,
selecting a level of parallelism and an operator fusion mechanism using the statistical information, and
optimizing the specialized intermediate representation code, based on the information of the runtime of performing the data search, to generate at least one optimized and compiled operator.
10 . The system of claim 9 , further comprising an operator specialization module that is operative to receive at least one operator file, as well as the runtime information from the query processing engine; and to have the at least one operator file optimized and compiled based on the runtime information of the data search.
11 . The system of claim 9 , wherein the at least one operator comprises annotations tagging functions that are to be specialized when the at least one operator is being optimized and compiled for the data search at the runtime of performing the data search.
12 . The system of claim 9 , wherein the information of the runtime of performing the data search further comprises:
data layout of the data search or data related thereto,
statistical information about the data search or data related thereto, or
a combination thereof.
13 . The system of claim 9 , wherein the optimization rules include global algorithms.
14 . The system of claim 9 , wherein the optimization rules are based on the runtime information of the runtime of performing the data search.
15 . The system of claim 9 , wherein the optimization rules include algorithms specific to the at least one operator.
16 . The system of claim 9 , wherein the performing the data search comprises generating optimized and compiled operators via the optimizing and compiling, which are subsequently executed by the data search according to a query plan defined by the data search.
17 . A non-transitory computer-readable media storing comprising computer instructions that configure at least one processor, upon execution of the instructions, to perform the following steps:
obtain intermediate representation code of at least one operator, the intermediate representation code configured in a high-level programming language and compiled by a front-end compiler;
optimized the intermediate representation code, based on information of the runtime of performing the data search;
compile the optimized intermediate representation code to generate at least one optimized and compiled operator; and
perform the data search by the query pr si engine using the at least one optimized and compiled operator;
wherein the information of the runtime of performing the data search comprises statistical information about the data search or the data related thereto; and
wherein the optimizing specializing, based on the information of the runtime of performing the data search the intermediate representation code for adapting to the data search and the hardware of the query processing engine,
selecting a level of parallelism and an operator fusion mechanism using the statistical information, and
optimizing the specialized intermediate representation code, based on the information of the runtime of performing the data search, to generate at least one optimized and compiled operator.
18 . The method of claim 1 , wherein the optimizing and compiling comprise:
obtaining the runtime information of the runtime of performing the data search;
obtaining optimization rules; and
one or a combination of operations selected from removing dead code in the intermediate representation code, removing dead branches in the representation code, and adding logic in the intermediate representation code to bypass unused portions of code based on the runtime information of the runtime of performing the data search and the optimization rules.
19 . The non-transitory computer-readable media of claim 17 , wherein the specializing the intermediate representation code comprises:
specializing, based on the runtime information of the runtime of performing the data search, the intermediate representation code by:
optimizing the at least one operator for the hardware of the query processing engine, and
defining or giving values to variables of the intermediate representation code that were previously unknown.
20 . The non-transitory computer-readable media of claim 17 , wherein the at least one operator comprises annotations tagging functions that are to be specialized when the at least one operator is being optimized and compiled for the data search at the runtime of performing the data search.
21 . The non-transitory computer-media of claim 17 , wherein the runtime information of the runtime of performing the data search further comprises:
data layout of the data search or data related thereto,
statistical information about the data search or data related thereto, or
a combination thereof.
22 . The non-transitory computer-readable media of claim 17 , wherein the optimization rules include global algorithms.
23 . The non-transitory computer-readable media of claim 17 , wherein the optimization rules are based on the runtime information of the runtime of performing the data search.
24 . The non-transitory computer-readable media of claim 17 , wherein the optimization rules include algorithms specific to the at least one operator.
25 . The non-transitory computer-readable media of claim 17 , wherein optimizing the at least one operator comprises an application of at least one optimization to a code of the at least one operator that increases the efficiency of a data search executing the at least one operator.
26 . The non-transitory computer-readable media of claim 17 , wherein performing a data search comprises the optimizing and compiling many operators, which are subsequently executed by the data search according to a query plan defined by the data search.