Techniques for generating pre-emptive expectation messages
Techniques a provided for performing multi-system operations in which changes are asynchronously committed in multiple systems. Metadata about the multi-system operation is injected into the commit logs of one system involved in a multi-system operation. An event stream is generated based on the commit logs of the one system, and is used to drive the operations that one or more other systems need to perform as part of the multi-system operation. A reconciliation system reads the logs of all systems involved in the multi-system operation and determines whether the multi-system operation completed successfully. Techniques are also provided for using machine learning to generate models of normal execution of different types of operations, detect anomalies, pre-emptively send expectation messages, and automatically suggest and/or apply fixes.
1. A method for identifying potential problems during a particular operation instance of a multi-system operation, comprising:
receiving a request, at a first service application associated with a first service, to initiate a particular operation instance of a particular type of multi-system operation;
the first service application generating log information indicating a first work performed by the first service application as part of the particular operation instance;
based on the log information and an expectation model associated with the particular type of multi-system operation, a pre-emptive warning system determining that, for the particular operation instance to complete normally, a particular action should be taken, within a particular time period, by a second service application associated with a second service;
performing pre-emptive expectation propagation by:
prior to expiration of the particular time period and before any anomaly has occurred during performance of the particular operation instance, the pre-emptive warning system sending an expectation message to the second service application;
wherein the expectation message indicates the particular action and timing information related to the particular time period;
based on the expectation message, the second service application detecting whether an anomaly has occurred during performance of the particular operation instance by determining whether the particular action is performed prior to expiration of the particular time period; and
responsive to detecting that the particular action was not performed prior to expiration of the particular time period, the second service application determining that an anomaly has occurred during performance of the particular operation instance and taking remedial action responsive to the occurrence of the anomaly.
2. The method of claim 1 further comprising: for the particular type of multi-system operation, training the expectation model to predict one or more actions that should occur within a time period for the particular operation instance of the particular type of multi-system operation to complete normally;
wherein training the expectation model comprises training the expectation model using a training set that is based on logs associated with previous executions of the particular type of multi-system operation.
3. The method of claim 2 wherein the training set includes information about calling graphs for the previous executions of the particular type of multi-system operation.
4. The method of claim 3 wherein the training set further includes information relating to amount of lapsed time between actions reflected in the calling graphs.
5. The method of claim 1 further comprising:
a streaming module generating an event stream based on the log information of the first service application;
wherein the event stream includes event records for multiple types of multi-system events;
the second service application subscribing to a subset of the types of multi-system events that are represented in the event stream of the first service application; and
wherein the subset includes the particular type of multi-system operation.
6. The method of claim 1 wherein the remedial action comprises generating an alert in response to non-occurrence of the particular action prior to expiration of the particular time period.
7. The method of claim 2 wherein:
the particular type of multi-system operation involves a plurality of services;
the method further comprises identifying the logs associated with previous executions of the particular type of multi-system operation by:
for each of the previous executions of the particular type of multi-system operation:
creating a unique identifier for an operation instance that corresponds to the previous execution; and
causing each service of the plurality of services to annotate its respective logs that are associated with the previous execution with the unique identifier created for the operation instance.
8. One or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, identify potential problems during a particular operation instance of a multi-system operation, the instructions causing:
receiving a request, at a first service application associated with a first service, to initiate a particular operation instance of a particular type of multi-system operation;
the first service application generating log information indicating a first work performed by the first service application as part of the particular operation instance;
based on the log information and an expectation model associated with the particular type of multi-system operation, a pre-emptive warning system determining that, for the particular operation instance to complete normally, a particular action should be taken, within a particular time period, by a second service application associated with a second service;
performing pre-emptive expectation propagation by:
prior to expiration of the particular time period and before any anomaly has occurred during performance of the particular operation instance, the pre-emptive warning system sending an expectation message to the second service application;
wherein the expectation message indicates the particular action and timing information related to the particular time period;
based on the expectation message, the second service application detecting whether an anomaly has occurred during performance of the particular operation instance by determining whether the particular action is performed prior to expiration of the particular time period; and
responsive to detecting that the particular action was not performed prior to expiration of the particular time period, the second service application determining that an anomaly has occurred during performance of the particular operation instance and taking remedial action responsive to the occurrence of the anomaly.
9. The one or more non-transitory computer-readable media of claim 8 further comprising instructions for:
for the particular type of multi-system operation, training the expectation model to predict one or more actions that should occur within a time period for the particular operation instance of the particular type of multi-system operation to complete normally;
wherein training the expectation model comprises training the expectation model using a training set that is based on logs associated with previous executions of the particular type of multi-system operation.
10. The one or more non-transitory computer-readable media of claim 9 wherein the training set includes information about calling graphs for the previous executions of the particular type of multi-system operation.
11. The one or more non-transitory computer-readable media of claim 10 wherein the training set further includes information relating to amount of lapsed time between actions reflected in the calling graphs.
12. The one or more non-transitory computer-readable media of claim 8 further comprising instructions for causing:
a streaming module generating an event stream based on the log information of the first service application;
wherein the event stream includes event records for multiple types of multi-system events;
the second service application subscribing to a subset of the types of multi-system events that are represented in the event stream of the first service application; and
wherein the subset includes the particular type of multi-system operation.
13. The one or more non-transitory computer-readable media of claim 8 wherein the remedial action comprises generating an alert in response to non-occurrence of the particular action prior to expiration of the particular time period.
14. The one or more non-transitory computer-readable media of claim 9 wherein:
the particular type of multi-system operation involves a plurality of services;
the one or more non-transitory computer-readable media further comprises instructions for identifying the logs associated with previous executions of the particular type of multi-system operation by:
for each of the previous executions of the particular type of multi-system operation:
creating a unique identifier for an operation instance that corresponds to the previous execution; and
causing each service of the plurality of services to annotate its respective logs that are associated with the previous execution with the unique identifier created for the operation instance.