Techniques for generating pre-emptive expectation messages
Techniques a provided for performing multi-system operations in which changes are asynchronously committed in multiple systems. Metadata about the multi-system operation is injected into the commit logs of one system involved in a multi-system operation. An event stream is generated based on the commit logs of the one system, and is used to drive the operations that one or more other systems need to perform as part of the multi-system operation. A reconciliation system reads the logs of all systems involved in the multi-system operation and determines whether the multi-system operation completed successfully. Techniques are also provided for using machine learning to generate models of normal execution of different types of operations, detect anomalies, pre-emptively send expectation messages, and automatically suggest and/or apply fixes.
1 . A method for identifying potential problems during a life-cycle of a particular object managed by a first service, comprising:
at the first service, maintaining a unique aggregate root identifier for each object of a plurality of objects;
wherein the plurality of objects share an object type;
wherein the plurality of objects includes the particular object;
during the life-cycle of each object of the plurality of objects, performing multiple multi-system operations that involve the particular object;
wherein each multi-system operation is reflected in log records of multiple systems that were involved in the multi-system operation;
wherein, in each of the multiple systems, each log record is associated with:
a correlation identifier that identifies a specific operation instance of the multi-system operation associated with the log record, and
an aggregate root identifier that identifies the object, of the plurality of objects, involved in the specific operation instance;
generating a correlation identifier and an aggregate root identifier for a non-legacy system of the multiple systems, wherein the correlation identifier and the aggregate root identifier corresponding to the first non-legacy system are generated by the non-legacy system;
generating a correlation identifier and an aggregate root identifier for a legacy system of the multiple systems, wherein the correlation identifier and the aggregate root identifier corresponding to the legacy system are generated by a system other than the legacy system;
in at least one of the multiple systems, injecting correlation-context metadata into a commit log as part of a database transaction by inserting the correlation-context metadata into a multi-system-operation table that is committed in the same transaction, thereby causing a streaming component to generate an event stream that includes the injected correlation-context metadata;
dynamically enriching the correlation-context metadata during execution of the multiple multi-system operations by augmenting the correlation-context metadata with attributes produced by respective systems as they perform portions of the multiple multi-system operations;
based on the log records associated with each aggregate root identifier, determining an expected life-cycle for objects that share the object type; and
based on the expected life-cycle and log records associated with the particular object, determining that an anomaly occurred during the life-cycle of the particular object.
2 . The method of claim 1 , wherein the step of generating a correlation identifier and an aggregate root identifier for the non-legacy system is performed as part of a logging operation.
3 . The method of claim 1 , wherein the step of generating a correlation identifier and an aggregate root identifier for the legacy system is performed as part of an annotation operation.
4 . The method of claim 3 , wherein the annotation operation is performed as an on-the-fly annotation operation that comprises:
detecting a first operation performed by the legacy system;
determining changes made by the first operation;
generating an annotation associated with the first operation.
5 . The method of claim 1 , wherein the expected life-cycle for objects that share the object type indicates that, after performance of a first type of multi-system operation, a second type of multi-system operation should be performed within a particular time period.
6 . The method of claim 5 , wherein determining that an anomaly occurred during the life-cycle of the particular object includes determining that the first type of multi-system operation was performed for the particular object, and the second type of multi-system operation was not performed for the particular object within a particular time period.
7 . The method of claim 1 , further comprising:
based on the expected life-cycle and log records associated with the particular object, determining an operation instance of the particular object that is associated with a delay in a life-cycle of the particular object.
8 . The method of claim 1 , further comprising:
in response to determining that the anomaly occurred, taking remedial action.
9 . The method of claim 8 , wherein taking remedial action comprises executing a compensation transaction that fixes the anomaly.
10 . The method of claim 8 , wherein taking remedial action comprises generating a fix suggestion regarding how to fix the anomaly.
11 . One or more non-transitory computer readable media comprising instructions which, when executed by one or more hardware processors, cause performance of operations for identifying potential problems during a life-cycle of a particular object managed by a first service, comprising:
at the first service, maintaining a unique aggregate root identifier for each object of a plurality of objects;
wherein the plurality of objects share an object type;
wherein the plurality of objects includes the particular object;
during the life-cycle of each object of the plurality of objects, performing multiple multi-system operations that involve the particular object;
wherein each multi-system operation is reflected in log records of multiple systems that were involved in the multi-system operation;
wherein, in each of the multiple systems, each log record is associated with:
a correlation identifier that identifies a specific operation instance of the multi-system operation associated with the log record, and
an aggregate root identifier that identifies the object, of the plurality of objects, involved in the specific operation instance;
generating a correlation identifier and an aggregate root identifier for a non-legacy system of the multiple systems, wherein the correlation identifier and the aggregate root identifier corresponding to the first non-legacy system are generated by the non-legacy system;
generating a correlation identifier and an aggregate root identifier for a legacy system of the multiple systems, wherein the correlation identifier and the aggregate root identifier corresponding to the legacy system are generated by a system other than the legacy system;
in at least one of the multiple systems, injecting correlation-context metadata into a commit log as part of a database transaction by inserting the correlation-context metadata into a multi-system-operation table that is committed in the same transaction, thereby causing a streaming component to generate an event stream that includes the injected correlation-context metadata;
dynamically enriching the correlation-context metadata during execution of the multiple multi-system operations by augmenting the correlation-context metadata with attributes produced by respective systems as they perform portions of the multiple multi-system operations;
based on the log records associated with each aggregate root identifier, determining an expected life-cycle for objects that share the object type; and
based on the expected life-cycle and log records associated with the particular object, determining that an anomaly occurred during the life-cycle of the particular object.
12 . The non-transitory computer readable media of claim 11 , wherein the step of generating a correlation identifier and an aggregate root identifier for the non-legacy system is performed as part of a logging operation.
13 . The non-transitory computer readable media of claim 11 , wherein the step of generating a correlation identifier and an aggregate root identifier for the legacy system is performed as part of an annotation operation.
14 . The non-transitory computer readable media of claim 13 , wherein the annotation operation is performed as an on-the-fly annotation operation that comprises:
detecting a first operation performed by the legacy system;
determining changes made by the first operation;
generating an annotation associated with the first operation.
15 . The non-transitory computer readable media of claim 11 , wherein the expected life-cycle for objects that share the object type indicates that, after performance of a first type of multi-system operation, a second type of multi-system operation should be performed within a particular time period.
16 . The non-transitory computer readable media of claim 15 , wherein determining that an anomaly occurred during the life-cycle of the particular object includes determining that the first type of multi-system operation was performed for the particular object, and the second type of multi-system operation was not performed for the particular object within a particular time period.
17 . The non-transitory computer readable media of claim 11 , wherein the operations further comprise:
based on the expected life-cycle and log records associated with the particular object, determining an operation instance of the particular object that is associated with a delay in a life-cycle of the particular object.
18 . The non-transitory computer readable media of claim 11 , wherein the operations further comprise:
in response to determining that the anomaly occurred, taking remedial action.
19 . The non-transitory computer readable media of claim 18 , wherein taking remedial action comprises executing a compensation transaction that fixes the anomaly.
20 . The non-transitory computer readable media of claim 18 , wherein taking remedial action comprises generating a fix suggestion regarding how to fix the anomaly.