Cost monitoring and cost-driven optimization of complex event processing system
View Patent ↗A cost monitoring system can monitor a cost of queries executing in a complex event processing system, running on top of a pay-as-you-go cloud infrastructure. Certain embodiments may employ a generic, cloud-platform independent cost model, multi-query optimization, cost calculation, and/or operator placement techniques, in order to monitor and explain query cost down to an operator level. Certain embodiments may monitor costs in near real-time, as they are created. Embodiments may function independent of an underlying complex event processing system and the underlying cloud platform. Embodiments can optimize a work plan of the cloud-based system so as to minimize cost for the end user, matching the cost model of the underlying cloud platform.
1. A computer-implemented method comprising:
a cost calculator implemented as a computer central processor component of a complex event processing (CEP) system indexing operators of a query received from a user;
the cost calculator component referencing a cost model including a memory cost, of the CEP system to assign an operator placement algorithm to the query;
a query optimization component of the CEP system performing a query optimization to generate an optimized query;
an operator placement component of the CEP system using the operator placement algorithm with the optimized query, to calculate a chosen operator placement;
the operator placement component deploying the optimized query onto a scalable streaming system;
the cost calculator component tracking and collecting run-time information regarding the optimized query on an operator level and on a query level from the scalable streaming system; and
the cost calculator component calculating tracked and collected costs of the optimized query and communicating the tracked and collected costs to the user.
2. A method as in claim 1 , the cost model further comprising a processing cost and a network cost.
3. A method as in claim 1 , the chosen operator placement calculated according to a bin packing approach.
4. A method as in claim 1 , the operator placement algorithm comprising:
an initial operator placement; and
a subsequent runtime operator placement.
5. A method as in claim 1 , the query optimization creating a global query graph.
6. A method as in claim 5 , the query optimization reusing common parts between the query and the global query graph.
7. A non-transitory computer readable storage medium embodying a computer program for performing a method, said method comprising:
a cost calculator component of a complex event processing (CEP) system indexing operators of a query received from a user;
the cost calculator component referencing a cost model including a memory cost, of the CEP system to assign an operator placement algorithm to the query;
a query optimization component of the CEP system performing a query optimization to generate an optimized query;
an operator placement component of the CEP system using the operator placement algorithm with the optimized query, to calculate a chosen operator placement;
the operator placement component deploying the optimized query onto a scalable streaming system;
the cost calculator component tracking and collecting run-time information regarding the optimized query on an operator level and on a query level from the scalable streaming system; and
the cost calculator component calculating tracked and collected costs of the optimized query and communicating the tracked and collected costs to the user.
8. A non-transitory computer readable storage medium as in claim 7 , the cost model further comprising a processing cost and a network cost.
9. A non-transitory computer readable storage medium as in claim 7 , the chosen operator placement calculated according to a bin packing approach.
10. A non-transitory computer readable storage medium as in claim 7 , the operator placement algorithm comprising:
an initial operator placement; and
a subsequent runtime operator placement.
11. A non-transitory computer readable storage medium as in claim 7 , the query optimization creating a global query graph.
12. A non-transitory computer readable storage medium as in claim 11 , the query optimization reusing common parts between the query and the global query graph.
13. A computer system comprising:
one or more hardware processors;
a software program, executable on a hardware processor of said computer system, the software program instructing:
a cost calculator component of a complex event processing (CEP) system, to index operators of a query received from a user;
the cost calculator component to reference a cost model including a memory cost, of the CEP system to assign an operator placement algorithm to the query;
a query optimization component of the CEP system to perform a query optimization to generate an optimized query;
an operator placement component of the CEP system to use the operator placement algorithm with the optimized query, to calculate a chosen operator placement;
the operator placement component to deploy the optimized query onto a scalable streaming system;
the cost calculator component to track and collect run-time information regarding the optimized query on an operator level and on a query level from the scalable streaming system; and
the cost calculator component to calculate tracked and collected costs of the optimized query and to communicate the tracked and collected costs to the user.
14. A computer system as in claim 13 , the cost model further comprising a processing cost and a network cost.
15. A computer system as in claim 13 , the chosen operator placement calculated according to a bin packing approach.
16. A computer system as in claim 13 , the operator placement algorithm comprising:
an initial operator placement; and
a subsequent runtime operator placement.
17. A computer system as in claim 13 , the query optimization creating a global query graph.
18. A computer system as in claim 17 , the query optimization reusing common parts between the query and the global query graph.