Granted Patent B2

US 9,804,951 · App. 14/800,679 · Granted Oct 31, 2017

Inventors: Phillip Liu (Palo Alto, CA); Arijit Mukherji (Fremont, CA); Rajesh Raman (Palo Alto, CA); Kris Grandy (San Carlos, CA); Jack Lindamood (San Mateo, CA)

Assignee: SignalFx, Inc.

G06F11/3644G06F11/3082G06F11/3466G06F2201/865G06F2201/88

Abstract

Claims (93)

1. A method for processing data streams generated by instrumented software, the method comprising:

receiving, a plurality of input data streams, each input data stream received from an instance of instrumented software executing on an external system, each data stream providing values of a metric, the values generated at variable time intervals;

receiving a request to periodically evaluate an expression based on data values of the input data streams;

for each input data stream, identifying a function for aggregating values of the metric of the input data stream;

generating a plurality of quantized data streams based on the input data streams, each quantized data stream comprising data values occurring periodically at a fixed time interval, the generating comprising, for each fixed time interval, for each input data stream, determining a data value of the quantized data stream for the fixed time interval, comprising:

determining a first aggregate value by applying the identified function over data values of the input data stream received within the fixed time interval;

determining a rollup time interval, wherein the rollup time interval is larger than the fixed time interval;

for each quantized data stream, determining a rollup data stream, the determining comprising:

determining a second aggregate value by applying the aggregation function to all data values of the quantized data stream generated within the rollup time interval; and

storing the second aggregate value as a value of the rollup data stream for the rollup time interval; and

periodically evaluating the expression based on data values of the rollup data streams.

2. The method of claim 1 , wherein the identified function is a first function, wherein the expression specifies a second function for aggregating values over the data streams, the evaluating of the expression comprising, repeatedly:

identifying a particular fixed time interval; and

applying the second function to the data values of the quantized data streams, each data value associated with the particular fixed time interval.

3. The method of claim 2 , wherein the expression determines aggregate values over groups of quantized data streams, each group associated with a value of a metadata attribute describing the data streams.

4. The method of claim 1 , wherein the data of an input data stream is generated by an instruction executed by the instrumented software, the instruction associated with one of a counter or a gauge.

5. The method of claim 1 , wherein the expression is evaluated for a fixed time interval, the method further comprising:

sending the result of evaluation of the expression for display as a chart updated for each fixed time interval.

6. The method of claim 1 , further comprising:

storing data values of a data stream received during a fixed time interval in a buffer; and

reusing the buffer for storing the data values of the data stream received during a subsequent fixed time interval.

7. The method of claim 1 , wherein the plurality of input data streams comprises a first input data stream and a second input data stream and a number of data values of a first input data stream received during a fixed time interval is different from a number of data values received for a second input data stream during the same fixed time interval.

8. The method of claim 1 , wherein a number of data values of an input data stream received during a fixed time interval is different from a number of data values received for the input data stream during a subsequent fixed time interval of the same length as the fixed time interval.

9. The method of claim 1 , wherein identifying the function for aggregating values of the metric of the input data stream comprises:

receiving information indicating that a metric for a data stream represents a count value; and

determining that the aggregation function for the data stream is a sum function.

10. The method of claim 1 , wherein identifying the function for aggregating values of the metric of the input data stream comprises:

receiving information indicating that a metric for a data stream represents a sum value; and

determining that the aggregation function for the data stream is a sum function.

11. The method of claim 1 , wherein identifying the function for aggregating values of the metric of the input data stream comprises:

receiving information indicating that a metric for a data stream represents a latest value; and

determining that the aggregation function for the data stream selects the latest value of a set of data values received in a time interval.

12. The method of claim 1 , wherein identifying the function for aggregating values of the metric of the input data stream comprises:

receiving information indicating that a metric for a data stream represents an average value; and

determining that the aggregation function for the data stream selects the latest value of a set of data values received in a time interval.

13. The method of claim 1 , wherein identifying the function for aggregating values of the metric of the input data stream comprises:

receiving information indicating that a metric for a data stream represents an average value, each data value comprising a tuple having the average value and a count of data values used for the average; and

determining that the aggregation function for the data stream for a time interval:

determines a product of each average value and count value received during the time interval,

determines a sum of the product values, and

determines a ratio of the sum of the product values and a sum of all count values.

14. The method of claim 1 , wherein identifying the function for aggregating values of the metric of the input data stream comprises:

receiving information indicating that a metric for a data stream represents an average value, each data value comprising a tuple having a sum of data values and a count of data values used for the sum; and

determining that the aggregation function for the data stream for a time interval:

determines a sum of all the sum values received during the time interval, and

determines a ratio of the sum values and a sum of all count values.

15. The method of claim 1 , wherein identifying the function for aggregating values of the metric of the input data stream comprises:

receiving information indicating that a metric for a data stream represents a maximum value; and

determining that the aggregation function for the data stream determines a maximum of data values received within a time interval.

16. The method of claim 1 , wherein identifying the function for aggregating values of the metric of the input data stream comprises:

receiving information indicating that a metric for a data stream represents a minimum value; and

determining that the aggregation function for the data stream determines a minimum of data values received within a time interval.

17. The method of claim 1 , wherein the expression determines one of: a percentile value, an average value, a minimum value, a maximum value, a sum value, or a moving average value.

18. A computer readable non-transitory storage medium storing instructions for processing data generated by instrumented software, the instructions when executed by a processor cause the processor to perform the steps of:

receiving, a plurality of input data streams, each input data stream received from an instance of instrumented software executing on an external system, each data stream providing values of a metric at variable time intervals;

receiving a request to periodically evaluate an expression based on the plurality of input data streams;

for each input data stream, identifying a function for aggregating values of the metric of the input data stream;

generating a plurality of quantized data streams based on the input data streams, each quantized data stream comprising data values occurring periodically at a fixed time interval, the generating comprising, for each fixed time interval, for each input data stream, determining a data value of the quantized data stream for the fixed time interval, comprising:

determining a first aggregate value by applying the identified function over data values of the input data stream received within the fixed time interval;

determining a rollup time interval, wherein the rollup time interval is larger than the fixed time interval;

for each quantized data stream, determining a rollup data stream, the determining comprising:

determining a second aggregate value by applying the aggregation function to all data values of the quantized data stream generated within the rollup time interval; and

storing the second aggregate value as a value of the rollup data stream for the rollup time interval; and

periodically evaluating the expression based on data values of the rollup data streams.

19. The computer readable non-transitory storage medium of claim 18 , wherein the identified function is a first function, wherein the expression specifies a second function for aggregating values over the data streams, the instructions for evaluating of the expression cause the processor to repeatedly perform the steps of:

identifying a particular fixed time interval; and

applying the second function to the data values of the quantized data streams, each data value associated with the particular fixed time interval.

20. The computer readable non-transitory storage medium of claim 19 , wherein the expression determines aggregate values over groups of quantized data streams, each group associated with a value of a metadata attribute describing the data streams.

21. The computer readable non-transitory storage medium of claim 18 , wherein the expression is evaluated for a fixed time interval, wherein the stored instructions when executed by the processor, further cause the processor to perform the steps of:

sending the result of evaluation of the expression for display as a chart updated for each fixed time interval.

22. The computer readable non-transitory storage medium of claim 18 , wherein the plurality of input data streams comprises a first input data stream and a second input data stream and a number of data values of a first input data stream received during a fixed time interval is different from a number of data values received for a second input data stream during the same fixed time interval.

23. The computer readable non-transitory storage medium of claim 18 , wherein a number of data values of an input data stream received during a fixed time interval is different from a number of data values received for the input data stream during a subsequent fixed time interval of the same length as the fixed time interval.

24. A computer-implemented system for processing data generated by instrumented software, the system comprising:

a computer processor; and

a computer readable non-transitory storage medium storing instructions thereon, the instructions when executed by a processor cause the processor to perform the steps of:

receiving, a plurality of input data streams, each input data stream received from an instance of instrumented software executing on an external system, each data stream providing values of a metric at variable time intervals;

receiving a request to periodically evaluate an expression based on the plurality of input data streams;

for each input data stream, identifying a function for aggregating values of the metric of the input data stream;

generating a plurality of quantized data streams based on the input data streams, each quantized data stream comprising data values occurring periodically at a fixed time interval, the generating comprising, for each fixed time interval, for each input data stream, determining a data value of the quantized data stream for the fixed time interval, comprising:

determining a first aggregate value by applying the identified function over data values of the input data stream received within the fixed time interval;

determining a rollup time interval, wherein the rollup time interval is larger than the fixed time interval;

for each quantized data stream, determining a rollup data stream, the determining comprising:

determining a second aggregate value by applying the aggregation function to all data values of the quantized data stream generated within the rollup time interval; and

storing the second aggregate value as a value of the rollup data stream for the rollup time interval; and

periodically evaluating the expression based on data values of the rollup data streams.

25. The computer-implemented system of claim 24 , wherein the identified function is a first function, wherein the expression specifies a second function for aggregating values over the data streams, the instructions for evaluating of the expression cause the processor to repeatedly perform the steps of:

identifying a particular fixed time interval; and

applying the second function to the data values of the quantized data streams, each data value associated with the particular fixed time interval.

26. The computer-implemented system of claim 25 , wherein the expression determines aggregate values over groups of quantized data streams, each group associated with a value of a metadata attribute describing the data streams.

27. The computer-implemented system of claim 24 , wherein the expression is evaluated for a fixed time interval, wherein the stored instructions when executed by the processor, further cause the processor to perform the steps of:

sending the result of evaluation of the expression for display as a chart updated for each fixed time interval.

28. The computer-implemented system of claim 24 , wherein the plurality of input data streams comprises a first input data stream and a second input data stream and a number of data values of a first input data stream received during a fixed time interval is different from a number of data values received for a second input data stream during the same fixed time interval.

29. The computer-implemented system of claim 24 , wherein a number of data values of an input data stream received during a fixed time interval is different from a number of data values received for the input data stream during a subsequent fixed time interval of the same length as the fixed time interval.

Assignments (8)

Continuity (4)