Online computer system with methodologies for distributed trace aggregation and for targeted distributed tracing
An online distributed computer system with methodologies for distributed trace aggregation and targeting distributed tracing. In one aspect, the disclosed distributed tracing technologies improve on existing distributed tracing technologies by providing to application developers and site operations personnel a more holistic and comprehensive insight into the behavior of the online distributed computer system in the form of computed span metric aggregates displayed in a graphical user interface thereby making it easier for such personnel to diagnose problems in the system and to support and maintain the system. In another aspect, the disclosed distributed tracing technologies improve on existing distributed tracing technologies by facilitating targeted tracing of initiator requests.
1. A method for initiating a targeted distributed trace in an online distributed computer system, the method comprising:
receiving, by an edge node of the online distributed computer system, a first trace query associated with a first targeted distributed trace, wherein the first targeted distributed trace includes tracing one or more interprocess communications (IPC) calls made, based on a received request, between nodes included in the online distributed computer system;
receiving, by the edge node, a first initiator request;
evaluating one or more fields included in the first initiator request against the first trace query;
determining that the first initiator request satisfies the first trace query when the one or more fields included in the first initiator request match at least a first portion of the first trace query;
generating first targeted trace data that is associated with the first targeted distributed trace of the one or more IPC calls, comprising:
generating a special trace event type value specifying that the first initiator request satisfies the first trace query, and
including, in the first targeted trace data, (i) the special trace event type value, and (ii) a trace identifier assigned to the first initiator request, wherein the first targeted trace data indicates that the first targeted distributed trace is associated with the first trace query; and
enabling the first targeted distributed trace of the one or more IPC calls within the online distributed computer system via the first targeted trace data by storing in thread local storage:
a field value indicating that tracing of the first initiator request is enabled, and
the trace identifier in association with the field value.
2. The method of claim 1 , wherein the first initiator request comprises a HyperText Transfer Protocol (HTTP) request.
3. The method of claim 1 , wherein generating the first targeted trace data comprises including the first trace query in the first targeted trace data.
4. The method of claim 1 , wherein enabling the first targeted distributed trace further comprises sending a first IPC call from the edge node to another node, wherein the first IPC call includes the field value and the trace identifier.
5. The method of claim 1 , wherein generating the first targeted trace data further comprises including, in the first targeted trace data, one or more fields of the first initiator request that satisfied the first trace query.
6. The method of claim 1 , wherein generating the first targeted trace data further comprises including, in the first targeted trace data, the first trace query.
7. The method of claim 1 , wherein the edge node receives the first trace query from a configuration server.
8. The method of claim 1 , wherein a value of the trace identifier is distinct from a value of the first trace query.
9. The method of claim 1 , wherein enabling the first targeted distributed trace further comprises generating a span identifier assigned to the first initiator request.
10. The method of claim 1 , further comprising:
receiving, by the edge node, a second trace query associated with tracing one or more IPC calls made, based on the received request, between the nodes included in the online distributed computer system, wherein:
the second trace query includes targeting criteria specifying one or more name-expression pairs included in the received request, and
a first name-expression pair is included in the second trace query and is not included in the first trace query;
evaluating one or more fields included in the first initiator request against the second trace query to determine whether the first initiator request includes at least one name-expression pair included in the targeting criteria; and
determining that the first initiator request does not satisfy the second trace query when the fields included in the first initiator request do not match at least one name-expression pair included in the first trace query.
11. One or more non-transitory computer-readable media storing program instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
receiving, by an edge node of an online distributed computer system, a first trace query associated with a first targeted distributed trace, wherein the first targeted distributed trace includes tracing one or more interprocess communications (IPC) calls made, based on a received request, between nodes included in the online distributed computer system;
receiving, by the edge node, a first initiator request;
evaluating one or more fields included in the first initiator request against the first trace query;
determining that the first initiator request satisfies the first trace query when the one or more fields included in the first initiator request match at least a first portion of the first trace query;
generating first targeted trace data that is associated with the first targeted distributed trace of the one or more IPC calls, comprising:
generating a special trace event type value specifying that the first initiator request satisfies the first trace query, and
including, in the first targeted trace data, (i) the special trace event type value, and (ii) a trace identifier assigned to the first initiator request, wherein the first targeted trace data indicates that the first targeted distributed trace is associated with the first trace query; and
enabling the first targeted distributed trace of the one or more IPC calls within the online distributed computer system via the first targeted trace data by storing in thread local storage:
a field value indicating that tracing of the first initiator request is enabled, and
the trace identifier in association with the field value.
12. An edge node computing device in an online distributed computer system, the edge node comprising:
a memory; and
a processor coupled to the memory that:
receives a first trace query associated with a first targeted distributed trace, wherein the first targeted distributed trace includes tracing one or more interprocess communications (IPC) calls made, based on a received request, between nodes included in the online distributed computer system;
receives a first initiator request;
evaluates one or more fields included in the first initiator request against the first trace query;
determines that the first initiator request satisfies the first trace query when the one or more fields included in the first initiator request match at least a first portion of the first trace query;
generates first targeted trace data that is associated with the first targeted distributed trace of the one or more IPC calls, comprising:
generating a special trace event type value specifying that the first initiator request satisfies the first trace query, and
including, in the first targeted trace data, (i) the special trace event type value, and (ii) a trace identifier assigned to the first initiator request, wherein the first targeted trace data indicates that the first targeted distributed trace is associated with the first trace query; and
enables the first targeted distributed trace of the one or more IPC calls within the online distributed computer system via the first targeted trace data by storing in thread local storage:
a field value indicating that tracing of the first initiator request is enabled, and
the trace identifier in association with the field value.
13. The edge node computing device of claim 12 , wherein the edge node sends the trace identifier in a first IPC call to another node when the first initiator request satisfies the first trace query.