Techniques for in-memory stitching
Event processing techniques for updating a database in real time based on events in a continuous event stream are disclosed. The techniques can update the database to incorporate information from thousands of received events per second. The events can include metrics measuring milestones for an organizational process defined by a user. Moreover, multiple streams can include metrics from many tenants concurrently. The techniques include receiving a plurality of events corresponding to a time interval and identifying that two events share a value of a key attribute. The techniques then include causing to be generated an in-memory combined database record comprising a combination of first and second attribute information from the two events and the value of the key attribute. In response to a trigger signal, the techniques include transmitting a single message, based at least in part on the combined database record, to update the database.
1. A method, comprising:
loading, by a computing device, a plurality of events corresponding to a time interval to an in-memory staging area implemented in a volatile memory of the computing device;
identifying, by the computing device, that a first event and a second event of the plurality of events in the in-memory staging area correspond to the time interval and share a value of a key attribute;
causing, by the computing device, a combined database record to be generated in the volatile memory of the computing device, the combined database record comprising a combination of at least one of a) attribute information from the first event or the second event, or b) the value of the key attribute; and
transmitting, to a database manager, a single message to update a database, the single message based at least in part on the combined database record, the single message to update the database being further based at least in part on a second combined database record, and the second combined database record comprising attribute information from other events corresponding to the time interval and sharing a second value of the key attribute.
2. The method of claim 1 , further comprising receiving a trigger signal, and wherein the transmitting is performed at least in response to receiving the trigger signal.
3. The method of claim 1 , wherein the single message comprises one or more of a database update statement, a database create statement, a database delete statement, or a database insert statement.
4. The method of claim 1 , wherein the single message comprises a Java Database Connectivity (JDBC) statement or a Hive statement.
5. The method of claim 1 , wherein causing the combined database record to be generated comprises applying a User Defined Aggregation Function (UDAF) or a custom function.
6. The method of claim 1 , wherein causing the combined database record to be generated comprises binding first attribute information from the first event and second attribute information together based at least in part on the first event and the second event sharing the value of the key attribute.
7. The method of claim 1 , wherein the plurality of events correspond to metrics of one or more workflows, wherein the one or more workflows are configured via a configuration user interface.
8. The method of claim 1 , further comprising:
generating a dashboard user interface comprising graphical controls configured to display statistics based at least in part on the updated database; and
providing the dashboard user interface to a client device for display.
9. A computing system, comprising:
one or more processors; and
a first memory configured to store computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to:
loading, by a computing device, a plurality of events corresponding to a time interval to an in-memory staging area implemented in a volatile memory of the computing device;
identifying, by the computing device, that a first event and a second event of the plurality of events in the in-memory staging area correspond to the time interval and share a value of a key attribute;
causing, by the computing device, a combined database record to be generated in the volatile memory of the computing device, the combined database record comprising a combination of at least one of a) attribute information from the first event or the second event, or b) the value of the key attribute; and
transmitting, to a database manager, a single message to update a database, the single message based at least in part on the combined database record, the single message to update the database being further based at least in part on a second combined database record, and the second combined database record comprising attribute information from other events corresponding to the time interval and sharing a second value of the key attribute.
10. The system of claim 9 , wherein the second memory of the computing device comprises at least one of a volatile memory or a non-volatile memory.
11. The system of claim 9 , wherein causing the combined database record to be generated comprises applying a User Defined Aggregation Function (UDAF) or a custom function.
12. The system of claim 9 , wherein causing the combined database record to be generated comprises binding first attribute information from the first event and second attribute information together based at least in part on the first event and the second event sharing the value of the key attribute.
13. A non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors of a computing device, cause the one or more processors to perform operations comprising:
loading, by the computing device, a plurality of events corresponding to a time interval to an in-memory staging area implemented in a volatile memory of the computing device;
identifying, by the computing device, that a first event and a second event of the plurality of events in the in-memory staging area correspond to the time interval and share a value of a key attribute;
causing, by the computing device, a combined database record to be generated in the volatile memory of the computing device, the combined database record comprising a combination of at least one of a) attribute information from the first event or the second event, or b) the value of the key attribute; and
transmitting, to a database manager, a single message to update a database, the single message based at least in part on the combined database record, the single message to update the database being further based at least in part on a second combined database record, and the second combined database record comprising attribute information from other events corresponding to the time interval and sharing a second value of the key attribute.
14. The non-transitory computer-readable medium of claim 13 , wherein the single message comprises one or more of a database update statement, a database create statement, a database delete statement, or a database insert statement.
15. The non-transitory computer-readable medium of claim 13 , wherein the single message comprises a Java Database Connectivity (JDBC) statement or a Hive statement.
16. The non-transitory computer-readable medium of claim 13 , wherein causing the combined database record to be generated comprises applying a User Defined Aggregation Function (UDAF) or a custom function.
17. The non-transitory computer-readable medium of claim 13 , wherein causing the combined database record to be generated comprises binding first attribute information from the first event and second attribute information together based at least in part on the first event and the second event sharing the value of the key attribute.