Identifying a root cause of an error
A process for analyzing an incident includes setting up an alert for a high error rate on a particular endpoint. Once the alert is triggered, a set of traces for transactions exhibiting errors on the offending endpoint is queried. All traces for other services/operations that include errors on the offending endpoint are also enumerated. A set of baseline transactions that involve the offending endpoint, but do not result in error may be utilized to determine whether the errors are always present, or are distinctive for certain offending transactions. All traces are ranked based on a statistic. Once the traces have been ranked, they may be traced down to a deepest/most terminal error. A set of transactions that correlate to the terminal error may also be analyzed to determine infrastructure causes.
1. A computer-implemented method for identifying a root cause of an error, the method comprising:
identifying a set of traces comprising at least an endpoint having an error rate above a threshold;
selecting a plurality of candidate error traces from the set of traces based on operations downstream from the endpoint;
ranking the plurality of candidate error traces;
identifying a root cause of the error based on the ranking of the plurality of candidate error traces;
training a machine-learning algorithm to identify the root cause of the error based at least in part on the ranking of the plurality of candidate error traces;
determining, based at least in part on the machine-learning algorithm, whether an infrastructure element correlating to the root cause of the error is contributing to the root cause of the error; and
causing for display a graphical representation of the root cause of the error, the graphical representation comprising a dashboard of a distributed microservices environment.
2. The computer-implemented method of claim 1 , wherein retrieving the set of traces including the endpoint that resulted in an error further comprises:
retrieving an error set and a baseline set from a datastore in response to an alert that the error rate exceeds the threshold, the error set comprising traces including the endpoint that resulted in an error, the baseline set comprising traces including the endpoint that executed successfully.
3. The computer-implemented method of claim 1 , wherein ranking the plurality of candidate error traces further comprises:
determining, for each trace, a first matrix of counts based on combinations of an error set and a baseline set, the error set comprising traces including the endpoint that resulted in an error, the baseline set comprising traces including the endpoint that executed successfully; and
ranking the first matrix of counts based on a statistic.
4. The computer-implemented method of claim 1 , further comprising:
calculating a sum of traces that include an error trace and the endpoint, in which an operation performed by the endpoint resulted in the error, to determine a first count;
calculating a sum of traces that include an error trace and the endpoint, in which an operation performed by the endpoint executed successfully, to determine a second count;
calculating a sum of traces that do not include an error trace and the endpoint, in which an operation performed by the endpoint resulted in the error, to determine a third count; and
calculating a sum of traces that do not include an error trace and the endpoint, in which an operation performed by the endpoint executed successfully, to determine a fourth count.
5. The computer-implemented method of claim 1 , further comprising:
determining that a probability that an endpoint results in an error, given an error trace is present, is higher than a probability that the endpoint results in the error given the error trace is not present.
6. The computer-implemented method of claim 1 , further comprising:
determining whether a baseline set is empty, the baseline set comprising traces including the endpoint that executed successfully; and
in response to determining that the baseline set is empty, sorting error traces by a proportion of an error set including a given error trace, the error set comprising traces including the endpoint that resulted in an error.
7. The computer-implemented method of claim 1 , wherein identifying the root cause of the error further comprises:
recording, on spans of the traces, an infrastructure element on which an execution occurred; and
for values of an infrastructure field, determining a second matrix of counts based on combinations of operations and infrastructure elements.
8. The computer-implemented method of claim 1 , further comprising:
ranking a second matrix of counts based on a statistic, the second matrix of counts based on combinations of operations and infrastructure elements; and
determining a terminal error of a span of an error trace based on the ranking of the second matrix of counts, the terminal error identified by tracing the span to its corresponding infrastructure element.
9. The computer-implemented method of claim 1 , further comprising determining a second matrix of counts, comprising:
calculating a sum of infrastructure fields that include operation errors to determine a fifth count;
calculating a sum of infrastructure fields that do not include operation errors to determine a sixth count;
calculating a sum of other infrastructure fields that do not include the infrastructure fields that include operation errors to determine a seventh count; and
calculating a sum of other infrastructure fields that do not include operation errors to determine an eighth count.
10. The computer-implemented method of claim 1 , further comprising:
determining a probability of an operation error given an infrastructure field is higher than a probability of an operation error given another infrastructure field.
11. A system for identifying a root cause of an error, the system comprising:
at least one memory having instructions stored thereon; and
at least one processor configured to execute the instructions, wherein the at least one processor is configured to:
identify a set of traces comprising at least an endpoint having an error rate above a threshold;
select a plurality of candidate error traces from the set of traces based on operations downstream from the endpoint;
rank the plurality of candidate error traces;
identify a root cause of the error based on the ranking of the plurality of candidate error traces;
train a machine-learning algorithm to identify the root cause of the error based at least in part on the ranking of the plurality of candidate error traces;
determine, based at least in part on the machine-learning algorithm, whether an infrastructure element correlating to the root cause of the error is contributing to the root cause of the error; and
cause for display a graphical representation of the root cause of the error, the graphical representation comprising a dashboard of a distributed microservices environment.
12. The system of claim 11 , wherein the processor is further configured to:
retrieve an error set and a baseline set from a datastore in response to an alert that the error rate exceeds the threshold, the error set comprising traces including the endpoint that resulted in an error, the baseline set comprising traces including the endpoint that executed successfully.
13. The system of claim 11 , wherein the processor is further configured to:
determine, for each trace, a first matrix of counts based on combinations of an error set and a baseline set, the error set comprising traces including the endpoint that resulted in an error, the baseline set comprising traces including the endpoint that executed successfully; and
rank the first matrix of counts based on a statistic.
14. The system of claim 11 , wherein the processor is further configured to:
calculate a sum of traces that include an error trace and the endpoint, in which an operation performed by the endpoint resulted in the error, to determine a first count;
calculate a sum of traces that include an error trace and the endpoint, in which an operation performed by the endpoint executed successfully, to determine a second count;
calculate a sum of traces that do not include an error trace and the endpoint, in which an operation performed by the endpoint resulted in the error, to determine a third count; and
calculate a sum of traces that do not include an error trace and the endpoint, in which an operation performed by the endpoint executed successfully, to determine a fourth count.
15. The system of claim 11 , wherein the processor is further configured to:
determine that a probability that an endpoint results in an error, given an error trace is present, is higher than a probability that the endpoint results in the error given the error trace is not present.
16. The system of claim 11 , wherein the processor is further configured to:
determine whether a baseline set is empty, the baseline set comprising traces including the endpoint that executed successfully; and
in response to determining that the baseline set is empty, sort error traces by a proportion of an error set including a given error trace, the error set comprising traces including the endpoint that resulted in an error.
17. The system of claim 11 , wherein the processor is further configured to:
record, on spans of the traces, an infrastructure element on which an execution occurred; and
for values of an infrastructure field, determine a second matrix of counts based on combinations of operations and infrastructure elements.
18. The system of claim 11 , wherein the processor is further configured to:
rank a second matrix of counts based on a statistic, the second matrix of counts based on combinations of operations and infrastructure elements; and
determine a terminal error of a span of an error trace based on the ranking of the second matrix of counts, the terminal error identified by tracing the span to its corresponding infrastructure element.
19. The system of claim 11 , wherein the processor is further configured to:
calculate a sum of infrastructure fields that include operation errors to determine a fifth count;
calculate a sum of infrastructure fields that do not include operation errors to determine a sixth count;
calculate a sum of other infrastructure fields that do not include the infrastructure fields that include operation errors to determine a seventh count; and
calculate a sum of other infrastructure fields that do not include operation errors to determine an eighth count.
20. A non-transitory computer-readable storage medium comprising instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform operations for identifying a root cause of an error, the operations comprising:
identifying a set of traces comprising at least an endpoint having an error rate above a threshold;
selecting a plurality of candidate error traces from the set of traces based on operations downstream from the endpoint;
ranking the plurality of candidate error traces;
identifying a root cause of the error based on the ranking of the plurality of candidate error traces;
training a machine-learning algorithm to identify the root cause of the error based at least in part on the ranking of the plurality of candidate error traces;
determining, based at least in part on the machine-learning algorithm, whether an infrastructure element correlating to the root cause of the error is contributing to the root cause of the error; and
causing for display a graphical representation of the root cause of the error, the graphical representation comprising a dashboard of a distributed microservices environment.