Tail-based span data sampling

1. A method for sampling span data, comprising, by a circuitry:

receiving, from agents of a host system, span data of operations performed by applications executed by the host system;

grouping the operations defined by the span data into traces;

determining a selected subset of the traces for span data sampling and an unselected subset of the traces excluded from the span data sampling according to a plurality of trace criteria, including by randomly selecting traces and including by comparing one or more trace characteristics of the traces with one or more trace criteria in the plurality of trace criteria, wherein:

the one or more trace characteristics include one or more of a trace duration, a trace error, or a trace priority level;

the one or more trace criteria compared to the one or more trace characteristics include one or more of a trace duration threshold, an occurrence of a trace error, or a trace priority level;

the selected subset of traces includes the randomly selected traces and one or more traces selected for: having a trace duration that exceeds the trace duration threshold, having an occurrence of trace error of interest or error type of interest, or having a trace priority level that exceed the trace priority level threshold;

prior to discarding unselected span data of the unselected subset of the traces, generating trace metrics using the unselected span data of the unselected subset of the traces and the selected span data of the selected subset of the traces, the trace metrics identifying the unselected subset of the traces, the selected subset of the traces, and a reasoning associated with each of the unselected subset of the traces and the selected subset of the traces, the reasoning indicating at least one of: random, duration, error, or priority; and

after generating the trace metrics, discarding the unselected span data of the unselected subset of the traces.

2. The method of claim 1 , further comprising, by the circuitry, providing selected span data of the selected subset of the traces to a server via a network.

3. The method of claim 1 , wherein the span data received from the agents includes all span data generated by the agents.

4. The method of claim 1 , wherein the applications include a web service, a registration service, and an account service.

5. The method of claim 1 , wherein the circuitry includes a plurality of accumulators that group the span data for the traces in parallel.

6. The method of claim 1 , wherein the span data includes a trace identifier for each operation and the operations are grouped into the traces using the trace identifier of each operation.

7. A system for sampling span data, comprising:

a circuitry configured to:

execute applications and agents, the agents configured to generate the span data of operations performed by the applications; and

execute a trace manager configured to:

receive the span data from the agents;

group the operations defined by the span data into traces;

determine a selected subset of the traces for span data sampling and an unselected subset of the traces excluded from the span data sampling according to a plurality of trace criteria, including by randomly selecting traces and including by comparing one or more trace characteristics of the traces with one or more trace criteria in the plurality of trace criteria, wherein:

the one or more trace characteristics include one or more of a trace duration, a trace error, or a trace priority level;

the one or more trace criteria compared to the one or more trace characteristics include one or more of a trace duration threshold, an occurrence of a trace error, or a trace priority level;

the selected subset of traces includes the randomly selected traces and one or more traces selected for: having a trace duration that exceeds the trade duration threshold, having an occurrence of trace error of interest or error type of interest, or having a trace priority level that exceeds the trace priority level threshold;

prior to discarding unselected span data of the unselected subset of the traces, generate trace metrics using the unselected span data of the unselected subset of the traces and the selected span data of the selected subset of the traces, the trace metrics identifying the unselected subset of the traces, the selected subset of the traces, and a reasoning associated with each of the unselected subset of the traces and the selected subset of the traces, the reasoning indicating at least one of: random, duration, error, or priority, and

after generating the trace metrics, discard the unselected span data of the unselected subset of the traces.

8. The system of claim 7 , wherein the trace manager is further configured to provide selected span data of the selected subset of the traces to a server via a network.

9. The system of claim 7 , wherein the span data received by the trace manager from the agents includes all span data generated by the agents.

10. The system of claim 7 , wherein the trace manager includes a plurality of accumulators executing in parallel, each accumulator configured to group the span data for a portion of the traces.

11. The system of claim 7 , wherein the span data includes a trace identifier for each operation and the operations are grouped into the traces using the trace identifier of each operation.

12. A non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, configures the processor to:

receive, from agents of a host system, span data of operations performed by applications executed by the host system;

group the operations defined by the span data into traces;

determine a selected subset of the traces for span data sampling and an unselected subset of the traces excluded from the span data sampling according to a plurality of trace criteria, including by randomly selecting traces and including by comparing one or more trace characteristics of the traces with one or more trace criteria in the plurality of trace criteria, wherein:

the one or more trace characteristics include one or more of a trace duration, a trace error, or a trace priority level;

the one or more trace criteria compared to the one or more trace characteristics include one or more of a trace duration threshold, an occurrence of trace error, or a trace priority level;

the selected subset of traces includes the randomly selected traces and one or more traces selected for: having a trace duration that exceeds the trace duration threshold, having an occurrence of trace error of interest or error type of interest, or having a trace priority level that exceeds the trace priority level threshold;

prior to discarding unselected span data of the unselected subset of the traces, generate trace metrics using the unselected span data of the unselected subset of the traces and the selected span data of the selected subset of the traces, the trace metrics identifying the unselected subset of the traces, the selected subset of the traces, and a reasoning associated with each of the unselected subset of the traces and the selected subset of the traces, the reasoning indicating at least one of: random, duration, error, or priority; and

after generating the trace metrics, discard the unselected span data of the unselected subset of the traces.