Elastic complex event processing
View Patent ↗Systems and methods according to embodiments provide elasticity for complex event processing (CEP) systems. Embodiments may comprise at least the following three components: (1) incremental query optimization, (2) operator placement, and (3) cost explanation. Incremental query optimization allows avoiding simultaneous computation of identical results by performing operator-level query reuse and subsumption. Using automatic operator placement, a centralized CEP engine can be transformed into a distributed one by dynamically distributing and adjusting the execution according to unpredictable changes in data and query load. Cost explanation functionality can provide end users with near real-time insight into the monetary cost of the whole system, down to operator level granularity. Combination of these components allows a CEP system to be scaled up and down.
1. A computer-implemented method comprising:
a cost explanation component maintaining a cost model, indexing a query submitted by a user;
the cost explanation component forwarding the query to a query optimization component maintaining a query graph from a standing query;
the query optimization component matching the query against the query graph to detect subsumption of operators, and produce a modified query graph;
forwarding the modified query graph to an operator placement component;
the operator placement component deploying a new filter operator based upon the modified query graph;
the cost explanation component receiving information regarding system utilization and operator performance statistics, from an associated distributed Complex Event Processing (CEP) system comprising a host upon which the new filter operator has been deployed;
based upon the information and the cost model, the cost explanation component providing feedback to the user regarding a monetary cost of the distributed CEP system.
2. A method as in claim 1 further comprising:
the operator placement component also receiving the information; and
the operator placement component using the information to perform run-time adaptation.
3. A method as in claim 1 , the operator placement component performs a two phase approach for operator placement comprising:
estimation; and
run-time monitoring and adaptation.
4. A method as in claim 1 , the host comprising an existing host of the distributed CEP system.
5. A method as in claim 1 , the host comprising a new host allocated from a resource pool.
6. A method as in claim 1 , the operator placement component deploying the new filter operator according to a bin packing technique.
7. A non-transitory computer readable storage medium embodying a computer program for performing a method, said method comprising:
a cost explanation component maintaining a cost model, indexing a query submitted by a user;
the cost explanation component forwarding the query to a query optimization component maintaining a query graph from a standing query;
the query optimization component matching the query against the query graph to detect subsumption of operators, and produce a modified query graph;
forwarding the modified query graph to an operator placement component;
the operator placement component deploying a new filter operator based upon the modified query graph;
the cost explanation component receiving information regarding system utilization and operator performance statistics, from an associated distributed Complex Event Processing (CEP) system comprising a host upon which the new filter operator has been deployed;
based upon the information and the cost model, the cost explanation component providing feedback to the user regarding a monetary cost of the distributed CEP system.
8. A non-transitory computer readable storage medium as in claim 7 , the method further comprising:
the operator placement component also receiving the information; and
the operator placement component using the information to perform run-time adaptation.
9. A non-transitory computer readable storage medium as in claim 7 , the operator placement component performing a two phase approach for operator placement comprising:
estimation; and
run-time monitoring and adaptation.
10. A non-transitory computer readable storage medium as in claim 7 , the host comprising an existing host of the distributed CEP system.
11. A non-transitory computer readable storage medium as in claim 7 , the host comprising a new host allocated from a resource pool.
12. A non-transitory computer readable storage medium as in claim 7 , the operator placement component deploying the new filter operator according to a bin packing technique.
13. A computer system comprising:
one or more hardware processors;
a software program, executable on a hardware processor of said computer system, the software program configured with:
a cost explanation component maintaining a cost model, to index a query submitted by a user;
the cost explanation component forwarding the query to a query optimization component maintaining a query graph from a standing query;
the query optimization component matching the query against the query graph to detect subsumption of operators, and produce a modified query graph;
forward the modified query graph to an operator placement component;
the operator placement component deploying a new filter operator based upon the modified query graph;
the cost explanation component receiving information regarding system utilization and operator performance statistics, from an associated distributed Complex Event Processing (CEP) system comprising a host upon which the new filter operator has been deployed;
based upon the information and the cost model, the cost explanation component providing feedback to the user regarding a monetary cost of the distributed CEP system.
14. A computer system as in claim 13 , the software program is further configured with:
the operator placement component also receiving the information; and
the operator placement component using the information to perform run-time adaptation.
15. A computer system as in claim 13 , the operator placement component performs a two phase approach for operator placement comprising:
estimation; and
run-time monitoring and adaptation.
16. A computer system as in claim 13 , the host comprising an existing host of the distributed CEP system.
17. A computer system as in claim 13 , the host comprising a new host allocated from a resource pool.
18. A computer system as in claim 13 , the operator placement component deploying the new filter operator according to a bin packing technique.