Operator sharing for continuous queries over archived relations
A continuous query identified by an archived relation of an application may be received. A physical query plan may be generated for the continuous query and/or a child operator plan may be identified. A buffer operator may be inserted between the child operator and its parent operator and/or the event information from the child operator may be stored in the buffer operator. A query operator may be identified in the physical query plan. It may be determined that the query operator identified in the physical query plan is a same type as a second query operator in a second continuous query concurrently executing. Further, a combined query plan may be generated based at least in part on the query operator identified in the physical query plan being the same type as the second query operator in the second continuous query.
1. A computer-implemented method, comprising:
identifying, by a computer system configured to execute computer-executable instructions, an archived relation associated with an application, the archived relation comprising a time-bounded window that is created with a combination of i) at least a portion of historical event data and ii) incoming events of a data stream associated with the application, the historical event data received from the data stream during a time preceding the creation of the archived relation;
receiving, by the computer system, a first continuous query, the first continuous query to be executed on the archived relation associated with the data stream that is associated with the application;
constructing a first archiver query for a first query operator configured to execute the first archiver query of the archived relation;
generating a first physical query plan for the first continuous query, the first physical query plan including a first child operator and a first parent operator, wherein the first child operator is the first query operator;
receiving, by the computer system, a second continuous query, the second continuous query concurrently executing at a same time on the archived relation as the first continuous query;
constructing a second archiver query for a second query operator configured to execute the second archiver query of the archived relation;
generating a second physical query plan for the second continuous query, the second physical query plan including the second query operator;
identifying the first query operator as a shareable query operator for sharing with the second query operator by determining that the first query operator and the second query operator are of a same type;
inserting a buffer operator between the shareable query operator and the first parent operator to be an output of the shareable query operator, the buffer operator storing event information related to the application to maintain an operator state for the first parent operator;
generating a combined archiver query comprising at least the first archiver query and the second archiver query;
generating a combined query plan based on the first physical query plan and the second physical query plan, the combined query plan comprising at least the buffer operator, the first parent operator, and the shareable query operator, the shareable query operator having all the outputs of both the first query operator and the second query operator;
executing the combined query plan for implementing the combined archiver query to obtain a result set of data records related to the application based at least in part on the shareable query operator;
initializing a buffer state of the buffer operator by constructing a list of tuples from the result set of data records, at least one tuple in the list of tuples being constructed based at least in part on a subset of attributes in the result set of data records;
generating a snapshot output of one or more data values related to the application based at least in part on the buffer state of the buffer operator; and
providing a dashboard user interface for presentation, the dashboard user interface configured to present the snapshot output.
2. The computer-implemented method of claim 1 , further comprising displaying the snapshot output of one or more data values related to the application to a user of the application via the dashboard user interface presented on a display device.
3. The computer-implemented method of claim 2 , further comprising processing incoming real-time events related to the application based at least in part on the snapshot output of the one or more data values.
4. The computer-implemented method of claim 1 , wherein identifying the archived relation further comprises loading the historical event data from a database of historical data.
5. The computer-implemented method of claim 1 , wherein the computer system is configured to manage a startup state associated with the archived relation being non-empty upon creation.
6. A non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by one or more computer systems, configure the one or more computer systems to perform instructions comprising:
identifying an archived relation associated with an application, the archived relation comprising a time-bounded window that is created with a combination of i) at least a portion of historical event data and ii) incoming events of a data stream associated with the application, the historical event data received from the data stream during a time preceding the creation of the archived relation;
receiving a first continuous query, the first continuous query to be executed on the archived relation associated with the data stream that is associated with the application;
constructing a first archiver query for a first query operator configured to execute the first archiver query of the archived relation;
generating a first physical query plan for the first continuous query, the first physical query plan including a first child operator and a first parent operator, wherein the first child operator is the first query operator;
receiving a second continuous query, the second continuous query concurrently executing at a same time on the archived relation as the first continuous query;
constructing a second archiver query for a second query operator configured to execute the second archiver query of the archived relation;
generating a second physical query plan for the second continuous query, the second physical query plan including the second query operator;
identifying the first query operator as a shareable query operator for sharing with the second query operator by determining that the first query operator and the second query operator are of a same type;
inserting a buffer operator between the shareable query operator and the first parent operator to be an output of the shareable query operator, the buffer operator storing event information related to the application to maintain an operator state for the first parent operator;
generating a combined archiver query comprising at least the first archiver query and the second archiver query;
generating a combined query plan based on the first physical query plan and the second physical query plan, the combined query plan comprising at least the buffer operator, the first parent operator, and the shareable query operator, the shareable query operator having all the outputs of both the first query operator and the second query operator;
executing the combined query plan for implementing the combined archiver query to obtain a result set of data records related to the application based at least in part on the shareable query operator;
initializing a buffer state of the buffer operator by constructing a list of tuples from the result set of data records, at least one tuple in the list of tuples being constructed based at least in part on a subset of attributes in the result set of data records;
generating a snapshot output of one or more data values related to the application based at least in part on the buffer state of the buffer operator; and
providing a dashboard user interface for presentation, the dashboard user interface configured to present the snapshot output.
7. A system, comprising:
a memory storing a plurality of instructions; and
a processor configured to access the memory, wherein the processor is further configured to execute the plurality of instructions to at least:
identify an archived relation comprising a time-bounded window that is created with a combination of i) at least a portion of historical event data and ii) incoming events of a data stream associated with the application, the historical event data received from the data stream during a time preceding the creation of the archived relation;
receive a first continuous query, the first continuous query to be executed on the archived relation associated with the data stream that is associated with the application;
construct a first archiver query for a first query operator configured to execute the first archiver query of the archived relation;
generate a first physical query plan for the first continuous query, the first physical query plan including a first child operator and a first parent operator, wherein the first child operator is the first query operator;
receive, by the computer system, a second continuous query, the second continuous query concurrently executing at a same time on the archived relation as the first continuous query;
construct a second archiver query for a second query operator configured to execute the second archiver query of the archived relation;
generate a second physical query plan for the second continuous query, the second physical query plan including the second query operator;
identify the first query operator as a shareable query operator for sharing with the second query operator by determining that the first query operator and the second query operator are of a same type;
insert a buffer operator between the shareable query operator and the first parent operator to be an output of the shareable query operator, the buffer operator storing event information related to the application to maintain an operator state for the first parent operator;
generate a combined archiver query comprising at least the first archiver query and the second archiver query;
generate a combined query plan based on the first physical query plan and the second physical query plan, the combined query plan comprising at least the buffer operator, the first parent operator, and the shareable query operator, the shareable query operator having all the outputs of both the first query operator and the second query operator;
execute the combined query plan for implementing the combined archiver query to obtain a result set of data records related to the application based at least in part on the shareable query operator;
initialize a buffer state of the buffer operator by constructing a list of tuples from the result set of data records, at least one tuple in the list of tuples being constructed based at least in part on a subset of attributes in the result set of data records
generate a snapshot output of one or more data values related to the application based at least in part on the buffer state of the buffer operator; and
provide a dashboard user interface for presentation, the dashboard user interface configured to present the snapshot output.
8. The system of claim 7 , wherein the processor is further configured to execute the plurality of instructions to at least display the snapshot output of the one or more data values related to the application to a user of the application via the dashboard user interface presented on a display device.