Techniques for in-memory stitching
Event processing techniques for updating a database in real time based on events in a continuous event stream are disclosed. The techniques can update the database to incorporate information from thousands of received events per second. The events can include metrics measuring milestones for an organizational process defined by a user. Moreover, multiple streams can include metrics from many tenants concurrently. The techniques include receiving a plurality of events corresponding to a time interval and identifying that two events share a value of a key attribute. The techniques then include causing to be generated an in-memory combined database record comprising a combination of first and second attribute information from the two events and the value of the key attribute. In response to a trigger signal, the techniques include transmitting a single message, based at least in part on the combined database record, to update the database.
1. A method, comprising:
loading, by a computing device, a plurality of events corresponding to a time interval to a staging area, the staging area implemented in a random access memory (RAM) of the computing device;
identifying, by the computing device, that a first event and a second event of the plurality of events in the staging area correspond to the time interval and share a value of a key attribute;
causing, by the computing device, a combined database record to be generated in the random access memory (RAM) of the computing device, the combined database record comprising a combination of at least:
attribute information from the first event or second event; or
the value of the key attribute; and
transmitting, to a database manager, a single message to update a database, the single message based at least in part on the combined database record.
2. The method of claim 1 , further comprising receiving a trigger signal, and wherein the transmitting is performed at least in response to receiving the trigger signal.
3. The method of claim 1 , wherein the single message comprises one or more of a database update statement, a database create statement, a database delete statement, or a database insert statement.
4. The method of claim 1 , wherein the single message comprises a Java Database Connectivity (JDBC) statement or a Hive statement.
5. The method of claim 1 , wherein the single message to update the database is further based at least in part on a second combined database record, the second combined database record comprising attribute information from other events corresponding to the time interval and sharing a second value of the key attribute.
6. The method of claim 1 , wherein causing the combined database record to be generated comprises applying a User Defined Aggregation Function (UDAF) or a custom function.
7. The method of claim 1 , wherein causing the combined database record to be generated comprises binding first attribute information from the first event and second attribute information together based at least in part on the first event and the second event sharing the value of the key attribute.
8. The method of claim 1 , wherein the plurality of events correspond to metrics of one or more workflows, wherein the one or more workflows are configured via a configuration user interface.
9. The method of claim 1 , further comprising:
generating a dashboard user interface comprising graphical controls configured to display statistics based at least in part on the updated database; and
providing the dashboard user interface to a client device for display.
10. A system, comprising:
one or more processors; and
a memory configured to store computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to:
load a plurality of events corresponding to a time interval to a staging area, the staging area implemented in a random access memory (RAM) of a computing device;
identify that a first event and a second event of the plurality of events correspond to the time interval and share a value of a key attribute;
cause to be generated a combined database record in the random access memory (RAM) of the computing device, the combined database record comprising a combination of at least:
attribute information from the first event or the second event; or
the value of the key attribute; and
transmit a single message to update a database, the single message based at least in part on the combined database record.
11. The system of claim 10 , wherein the single message comprises one or more of a database update statement, a database create statement, a database delete statement, or a database insert statement.
12. The system of claim 10 , wherein the single message comprises a Java Database Connectivity (JDBC) statement or a Hive statement.
13. The system of claim 10 , wherein the single message to update the database is further based at least in part on a second combined database record, the second combined database record comprising attribute information from other events corresponding to the time interval and sharing a second value of the key attribute.
14. The system of claim 10 , wherein causing the combined database record to be generated comprises applying a User Defined Aggregation Function (UDAF) or a custom function.
15. The system of claim 10 , wherein causing the combined database record to be generated comprises binding first attribute information from the first event and second attribute information together based at least in part on the first event and the second event sharing the value of the key attribute.
16. A non-transitory computer-readable medium storing computer-executable instructions that, when executed by a processor, cause the processor to perform operations comprising:
loading a plurality of events corresponding to a time interval to a staging area, the staging area implemented in a random access memory (RAM) of a computing device;
identifying that a first event and a second event of the plurality of events correspond to the time interval and share a value of a key attribute;
causing to be generated a combined database record in the random access memory (RAM) of the computing device, the combined database record comprising a combination of at least:
attribute information from the first event or the second event; or
the value of the key attribute; and
transmitting a single message to update a database, the single message based at least in part on the combined database record.
17. The non-transitory computer-readable medium of claim 16 , wherein the single message comprises one or more of a database update statement, a database create statement, a database delete statement, or a database insert statement.
18. The non-transitory computer-readable medium of claim 16 , wherein the single message comprises a Java Database Connectivity (JDBC) statement or a Hive statement.
19. The non-transitory computer-readable medium of claim 16 , wherein causing the combined database record to be generated comprises applying a User Defined Aggregation Function (UDAF) or a custom function.
20. The non-transitory computer-readable medium of claim 16 , wherein causing the combined database record to be generated comprises binding first attribute information from the first event and second attribute information together based at least in part on the first event and the second event sharing the value of the key attribute.