Rule-based extendable query optimizer

1. A method comprising:

receiving a query;

generating an initial data flow graph comprising a plurality of nodes for executing the query;

optimizing the initial data flow graph using a model optimizer, the model optimizer comprising an optimizer framework and an application programming interface (API), the optimizer framework providing logic to restructure the initial data flow graph and a rules engine for executing one or more optimization rules, the API allowing for registration of new optimization rules to be executed by the rules engine;

initiating executing of the query using the optimized data flow graph, wherein each rule identifies a start node in the initial data flow graph from which optimization can be initiated;

subsequently accessing a second subset of the plurality of patterns to second identify a matching second pattern N and executing at least one first optimization rule associated with the matching second pattern, the second subset of the plurality of patterns comprising different patterns than a first subset of the plurality of patterns, wherein N is a number;

repeatedly checking pattern N+1 based on the pattern N that was previously a match and corresponding rule N, wherein each pattern specifies how many preceding and succeeding nodes for determining whether the pattern matches;

assigning priorities to at least a portion of the optimization rules, the priorities being used to determine when to execute a particular optimization rule, wherein the optimizer framework assigns cost functions to at least a portion of the optimization rules, the cost functions being used to determine when to execute a particular optimization rule, wherein each rule identifies one to N predecessors of the corresponding start node from which optimization can be initiated, wherein each rule specifies read and write operations to perform on corresponding nodes; and

providing a list of nodes of the graph to be removed by optimizer framework during optimizing via the API.

2. The method as in claim 1 , wherein at least one optimizer rule used by the model optimizer affects only a single node of the initial data graph during optimizing.

3. The method as in claim 1 , wherein at least one optimizer rule used by the model optimizer affects at least two nodes of the initial data graph during optimizing.

4. The method as in claim 1 , wherein the optimizer framework assigns priorities to at least a portion of the optimization rules, the priorities being used to determine when to execute a particular optimization rule.

5. The method as in claim 1 , wherein the receiving, generating, optimizing, and initiating are executed by one or more data processors across one or more computing systems.

6. An article of manufacture comprising:

computer executable instructions stored on non-transitory computer readable media, which, when executed by at least one data processor, causes the at least one data processor to perform operations comprising:

receiving a query;

generating an initial data flow graph comprising a plurality of nodes for executing the query;

optimizing the initial data flow graph using a model optimizer, the model optimizer comprising an optimizer framework and an application programming interface (API), the optimizer framework providing logic to restructure the initial data flow graph and a rules engine for executing one or more optimization rules, the API allowing for registration of new optimization rules to be executed by the rules engine;

initiating executing of the query using the optimized data flow graph, wherein each rule identifies a start node in the initial data flow graph from which optimization can be initiated;

subsequently accessing a second subset of the plurality of patterns to second identify a matching second pattern N and executing at least one first optimization rule associated with the matching second pattern, the second subset of the plurality of patterns comprising different patterns than a first subset of the plurality of patterns, wherein N is a number;

repeatedly checking pattern N+1 based on the pattern N that was previously a match and corresponding rule N, wherein each pattern specifies how many preceding and succeeding nodes for determining whether the pattern matches;

assigning priorities to at least a portion of the optimization rules, the priorities being used to determine when to execute a particular optimization rule, wherein the optimizer framework assigns cost functions to at least a portion of the optimization rules, the cost functions being used to determine when to execute a particular optimization rule, wherein each rule identifies one to N predecessors of the corresponding start node from which optimization can be initiated, wherein each rule specifies read and write operations to perform on corresponding nodes; and

providing a list of nodes of the graph to be removed by optimizer framework during optimizing via the API.

7. The article as in claim 6 , wherein at least one optimizer rule used by the model optimizer affects only a single node of the initial data graph during optimizing.

8. The article as in claim 6 , wherein at least one optimizer rule used by the model optimizer affects at least two nodes of the initial data graph during optimizing.

9. The article as in claim 6 , wherein the optimizer framework assigns priorities to at least a portion of the optimization rules, the priorities being used to determine when to execute a particular optimization rule.

10. A system comprising:

one or more data processors forming part of at least one computing system; and

memory coupled to the at least one data processors storing instructions, which when executed, result in operations comprising:

receiving a query;

generating an initial data flow graph comprising a plurality of nodes for use by a calculation engine in executing the query, each node defining one or more operations for execution by a database server;

optimizing the initial data flow graph using a model optimizer, the model optimizer comprising an optimizer framework and an application programming interface (API), the optimizer framework providing logic to restructure the initial data flow graph and a rules engine for executing one or more optimization rules, the API allowing for registration of new optimization rules to be executed by the rules engine;

initiating executing of the query by instantiating the optimized data flow graph for use by the calculation engine, wherein each rule identifies a start node in the initial data flow graph from which optimization can be initiated;

subsequently accessing a second subset of the plurality of patterns to second identify a matching second pattern N and executing at least one first optimization rule associated with the matching second pattern, the second subset of the plurality of patterns comprising different patterns than a first subset of the plurality of patterns, wherein N is a number;

repeatedly checking pattern N+1 based on the pattern N that was previously a match and corresponding rule N, wherein each pattern specifies how many preceding and succeeding nodes for determining whether the pattern matches;

assigning priorities to at least a portion of the optimization rules, the priorities being used to determine when to execute a particular optimization rule, wherein the optimizer framework assigns cost functions to at least a portion of the optimization rules, the cost functions being used to determine when to execute a particular optimization rule, wherein each rule identifies one to N predecessors of the corresponding start node from which optimization can be initiated, wherein each rule specifies read and write operations to perform on corresponding nodes; and

providing a list of nodes of the graph to be removed by optimizer framework during optimizing via the API.