Ranking filter algorithms
A method, a system, and a computer program product for executing a query. A query associated with a calculation scenario defining a data flow model that includes one or more calculation nodes is received. Each calculation node corresponds to an operation being performed on one or more database tables stored at a database. The calculation nodes include a first calculation node corresponding to a ranking filter operation. A calculation engine executes the calculation scenario and performs the ranking filter operation to generate a data structure storing a plurality of rows of the database tables. Based on the executing of the calculation scenario, a result data set including at least one row from the plurality of rows is generated and provided by a database server to an application server.
1. A computer-implemented method, comprising:
receiving, by a database server from an application server, a query associated with a calculation scenario defining a data flow model that comprises one or more calculation nodes, each of the one or more calculation nodes corresponding to an operation being performed on one or more database tables stored at a database, the one or more calculation nodes comprising a first calculation node corresponding to a ranking filter operation;
executing, by a calculation engine, the calculation scenario by performing the ranking filter operation to generate a data structure storing a plurality of rows of the one or more database tables responsive to the query, wherein the data structure comprises a mapping of each row in the plurality of rows to partitions of the one or more database tables processed by the calculation engine during execution of the calculation scenario including the ranking filter operation, wherein the ranking filter operation comprises an aggregation operation and a filter operation being performed together and configured to implement dynamic ordering and partitioning;
generating, based on the executing of the calculation scenario and using the data structure, a result data set comprising at least one row from the plurality of rows, wherein generating the result data set comprises:
splitting the one or more database tables into parts,
generating, for each of the parts, separate data structures storing partial data results, and
combining the partial data results from each of the parts,
detecting, by a persistence layer, a restart of the application server associated with a crash of a network-attached storage system of the application server;
determining a portion of the result data set associated with unfinished transactions; and
providing, by the database server to the application server, the result data set excluding the portion of the result data set associated with the unfinished transactions that are identified using log buffers reordered from inconsecutive log segments of the partitions during a recovery process executed after the crash of the network-attached storage system of the application server.
2. The computer-implemented method of claim 1 , wherein generating further comprises selecting the at least one row in the result data set from the plurality of rows based on the mapping.
3. The computer-implemented method of claim 2 , wherein at least one partition in the partitions in the mapping is configured to map to another data structure storing one or more rows for inclusion in the result data set.
4. The computer-implemented method of claim 3 , wherein the result data set comprises a predetermined number of rows.
5. The computer-implemented method of claim 3 , wherein the one or more rows in the another data structure are selected based on one or more ordering attributes defining a predetermined order of execution of operations of the received query on the partitions of the one or more database tables.
6. The computer-implemented method of claim 1 , wherein the calculation engine invokes an SQL processor for executing set operations.
7. The computer-implemented method of claim 1 , wherein an input for each calculation node comprises one or more of: a physical index, a join index, an OLAP index, and another calculation node.
8. The computer-implemented method of claim 7 , wherein each calculation node has at least one output table that is used to generate the result data set.
9. The computer-implemented method of claim 8 , wherein at least one calculation node consumes an output table of another calculation node.
10. The computer-implemented method of claim 1 , wherein the database is a column-oriented in-memory database.
11. A system comprising:
at least one programmable processor; and
a non-transitory machine-readable medium storing instructions that, when executed by the at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
receiving, by a database server from an application server, a query associated with a calculation scenario defining a data flow model that comprises one or more calculation nodes, each of the one or more calculation nodes corresponding to an operation being performed on one or more database tables stored at a database, the one or more calculation nodes comprising a first calculation node corresponding to a ranking filter operation;
executing, by a calculation engine, the calculation scenario by performing the ranking filter operation to generate a data structure storing a plurality of rows of the one or more database tables responsive to the query, wherein the data structure comprises a mapping of each row in the plurality of rows to partitions of the one or more database tables processed by the calculation engine during execution of the calculation scenario including the ranking filter operation, wherein the ranking filter operation comprises an aggregation operation and a filter operation being performed together and configured to implement dynamic ordering and partitioning;
generating, based on the executing of the calculation scenario and using the data structure, a result data set comprising at least one row from the plurality of rows, wherein generating the result data set comprises:
splitting the one or more database tables into parts,
generating, for each of the parts, separate data structures storing partial data results, and
combining the partial data results from each of the parts,
detecting, by a persistence layer, a restart of the application server associated with a crash of a network-attached storage system of the application server;
determining a portion of the result data set associated with unfinished transactions; and
providing, by the database server to the application server, the result data set excluding the portion of the result data set associated with the unfinished transactions that are identified using log buffers reordered from inconsecutive log segments of the partitions during a recovery process executed after the crash of the network-attached storage system of the application server.
12. The system of claim 11 , wherein generating further comprises selecting the at least one row in the result data set from the plurality of rows based on the mapping.
13. The system of claim 12 , wherein at least one partition in the partitions in the mapping is configured to map to another data structure storing one or more rows for inclusion in the result data set.
14. The system of claim 13 , wherein the result data set comprises a predetermined number of rows.
15. The system of claim 13 , wherein the one or more rows in the another data structure are selected based on one or more ordering attributes defining a predetermined order of execution of operations of the query on the partitions of the one or more database tables.
16. A computer program product comprising a non-transitory machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
receiving, by a database server from an application server, a query associated with a calculation scenario defining a data flow model that comprises one or more calculation nodes, each of the one or more calculation nodes corresponding to an operation being performed on one or more database tables stored at a database, the one or more calculation nodes comprising a first calculation node corresponding to a ranking filter operation;
executing, by a calculation engine, the calculation scenario by performing the ranking filter operation to generate a data structure storing a plurality of rows of the one or more database tables responsive to the query, wherein the data structure comprises a mapping of each row in the plurality of rows to partitions of the one or more database tables processed by the calculation engine during execution of the calculation scenario including the ranking filter operation, wherein the ranking filter operation comprises an aggregation operation and a filter operation being performed together and configured to implement dynamic ordering and partitioning;
generating, based on the executing of the calculation scenario and using the data structure, a result data set comprising at least one row from the plurality of rows, wherein generating the result data set comprises:
splitting the one or more database tables into parts,
generating, for each of the parts, separate data structures storing partial data results, and
combining the partial data results from each of the parts,
detecting, by a persistence layer, a restart of the application server associated with a crash of a network-attached storage system of the application server;
determining a portion of the result data set associated with unfinished transactions; and
providing, by the database server to the application server, the result data set excluding the portion of the result data set associated with the unfinished transactions that are identified using log buffers reordered from inconsecutive log segments of the partitions during a recovery process executed after the crash of the network-attached storage system of the application server.