Optimized unbiased statistical analysis of partially sampled traces without completeness information
A technology is disclosed for maximizing the creation of transaction trace data by multiple, different monitoring data sources like agents having individual volume constraints for created trace data. Trace context data identifying individual transactions and containing shared randomness data is propagated between agents and used in created trance data to maintain transaction identity in trace data fragments and for consistent sampling decisions. Sampling decisions for individual trace data fragments are based on the shared randomness data and on an agent-autonomously defined sampling probability. Values of randomness data and sampling probability are restricted to a limited number, like the values of a geometric series with a common ratio of 1/2. Shared randomness data and sampling probability are included in created trace data. Restricting randomness data and sampling probability to values of a geometric series with common ratio 1/2 leads to additional numeric advantages for the computer implemented calculation of estimation results.
1 . A computer-implemented method for reporting transaction trace data for a computer transaction executing in a distributed computing environment, comprising:
receiving, by an agent, span data from a sensor instrumented in a given method executed by a monitored computer transaction, where the span data describes a portion of execution of the monitored computer transaction performed by the given method and includes a unique identifier for the monitored computer transaction;
retrieving, by the agent, a shared sampling number for the monitored computer transaction from a data store;
randomly selecting, by the agent, a value for the shared sampling number and storing the value for the shared sampling number in response to the shared sampling number not being present in the data store, where value for the shared sampling number is randomly selected from a limited set of values;
detecting, by the agent, an event of the monitored computer transaction that crosses an execution boundary of a thread, a process or a host computing system and making the unique identifier and the shared sampling number for the monitored computer transaction accessible to other agents in response to detecting said event;
determining, by the agent, a sampling probability for the span data, where the sampling probability defines a percentage of span data reported by the agent and a value for the sampling probability is selected from the limited set of values;
comparing, by the agent, the shared sampling number to the sampling probability;
appending, by the agent, the sampling probability to the span data; and
sending, by the agent, the span data as a sampled span data record via a network to a monitoring server, where the sampled span data record is sent to the monitoring server in response to the shared sampling number being less than the sampling probability.
2 . The method of claim 1 further comprises discarding, by the agent, the span data in response to the shared sampling number being greater than or equal to the sampling probability.
3 . The method of claim 1 wherein each value in the limited set of values is greater than zero, smaller than or equal to one and where a given value in the limited set of values is a multiple of another given value in the limited set of values.
4 . The method of claim 1 wherein each value in the limited set of values is a reciprocal of a power of two.
5 . The method of claim 1 wherein transaction trace data is a set of sampled span data record, each sampled span data record includes the unique identifier for the monitored computer transaction, a unique identifier for the given method, a sampling probability determined by the agent, and observation data for a given metric describing execution of the given method, and each sampled span data record in the set of sampled span data records has same unique identifier for the monitored computer transaction.
6 . The method of claim 5 further comprises adjusting, by the agent, the sampling probability for the span data based on computing resources available on the computing device hosting the agent.
7 . The method of claim 5 further comprises adjusting, by the agent, the sampling probability for the span data based on type of method associated with the set of span data records.
8 . The method of claim 5 further comprises detecting, by the agent, an undesired execution outcome and adjusting the sampling probability for the span data in response to detecting the undesired execution outcome.
9 . The method of claim 1 further comprises
maintaining, by the agent, a unique identifier for last sampled span data record sent to the monitoring server in the data store;
maintaining, by the agent, a counter indicating number of span data not reported to the monitoring server in the data store;
discarding, by the agent, span data and incrementing the counter by one in response to the shared sampling number being greater than or equal to the sampling probability;
creating a sampled span data record from the span data, where the sampled span data record includes unique identifier for last span sent to the monitoring server and the counter value, where the sampled span data record is created in response to the shared sampling number being less than the sampling probability.
10 . The method of claim 9 further comprises setting the unique identifier for the last sampled span data record to an identifier for current span data and setting the counter to zero in response to the shared sampling number being less than the sampling probability.
11 . The method of claim 1 further comprises
receiving, by the agent, a desired sampling rate;
identifying a first sampling probability from the limited set of values, where the first sampling probability is closest value in the limited set of values that is smaller than the desired sampling rate;
identifying a second sampling probability from the limited set of values, where the second sampling probability is closest value in the limited set of values that is larger than the desired sampling rate;
performing a sampling decision for a plurality of sampled span data records using the first sampling probability and the second sampling probability, where the sampling decision randomly selects either the first or the second sampling probability, such that the desired sampling rate is achieved for the plurality of sampled span data records.
12 . The method of claim 5 wherein sending the span data further comprises storing the sampled span data records in a buffer on the computing device hosting the agent, periodically fetching the stored sampled span data records from the buffer and sending the fetched sampled span data records to the monitoring server.
13 . The method of claim 11 further comprises
appending, by the agent, the shared sampling number to the span data;
receiving, by the agent, a new sampled span data record;
in response to the buffer being full, selecting, by the agent, a given sampled span data record stored in the buffer and having highest shared sampling number;
comparing, by the agent, shared sampling number associated with the new sampled span data record to the shared sampling number from the given span data record;
replacing, by the agent, the given sampled span data record in the buffer with the new sampled span data record in response to the shared sampling number associated with the new sampled span data record being larger than the shared sampling number from the given sampled span data record; and
discarding, by the agent, the new sampled span data record in response to the shared sampling number associated with the new sampled span data record being smaller than the shared sampling number from the given sampled span data record.
14 . The method of claim 11 further comprises
appending, by the agent, the shared sampling number to the span data;
receiving, by the agent, a new sampled span data record, in response to the buffer being full;
b) randomly selecting, by the agent, a given sampled span data record stored in the buffer, where the given span data record has lowest sampling probability;
c) comparing, by the agent, sampling probability associated with the new sampled span data record to the sampling probability from the given sampled span data record;
d) replacing, by the agent, the given sampling span data record in the buffer with the new sampled span data record in response to the sampling probability associated with the sampled new span data record being larger than the sampling probability from the given sampled span data record;
e) updating, by the agent, the sampling probability associated with the new sampled span data record;
f) comparing, by the agent, the sampling probability associated with the new sampled span data record to shared sampling number of the new sampled span data record; and
repeating steps b)-f) in response to the sampling probability associated with the new span data record being less than the shared sampling number of new sampled span data record.
15 . The method of claim 5 further comprises
receiving, by the agent, a new span data record;
determining, by the agent, a current elapsed time between receiving the new span data record and the span data record most recently received by the agent;
calculating an estimate for the average elapsed time between receipt of span data records by aggregating the current elapsed time with previously observed elapsed times between receipt of span data records; and
determining, by the agent, a sampling probability for the new span data record in part based on the estimated average elapsed time such that magnitude of the sampling probability correlates inversely with the estimated average elapsed time.