Representing result data streams based on execution of data stream language programs
An instrumentation analysis system processes data streams by executing instructions specified using a data stream language program. The data stream language allows users to specify a search condition using a find block for identifying the set of data streams processed by the data stream language program. The set of identified data streams may change dynamically. The data stream language allows users to group data streams into sets of data streams based on distinct values of one or more metadata attributes associated with the input data streams. The data stream language allows users to specify a threshold block for determining whether data values of input data streams are outside boundaries specified using low/high thresholds. The elements of the set of data streams input to the threshold block can dynamically change. The low/high threshold values can be specified as data streams and can dynamically change.
1. A method for generating dynamic real-time reports using data streams generated by a data stream language program, the method comprising:
receiving data from a plurality of data streams, each data stream generated by an instance of instrumented software executing on a system;
storing metadata describing the plurality of data streams, the metadata for each data stream including one or more attributes associated with the data stream;
receiving a set of instructions specified using a data stream language program for processing the plurality of data streams, the instructions comprising one or more grouping commands for grouping data streams, each grouping command associated with one or more attributes of data streams and an aggregation operation, and the instructions comprising a publish block, the publish block associated with a metric name used as an attribute;
generating one or more result data streams by executing the one or more grouping commands to group data derived from the data streams by a set of the one or more attributes of the data streams;
identifying the last grouping command executed in the set of instructions;
determining a set of attributes associated with the last grouping command as a subset of the set of the one or more attributes of the data streams;
for each of the one or more result data streams:
determining a distinct subset of the set of attributes associated with the last grouping command, wherein the distinct subset is used by the last grouping command in generating the result data stream;
determining the values of the attributes in the distinct subset;
concatenating the determined values of the distinct subset;
storing the concatenated values as first metadata describing the result data stream;
storing the metric name of the publish block as second metadata describing the result data stream;
generating an identifier for the result data stream using the first and second stored metadata describing the result data stream;
storing the result data stream in association with the identifier; and
automatically generating a real-time report identified by the generated identifier and containing the stored result data stream.
2. The method of claim 1 , further comprising:
using the data of the result data stream as input to another block of another data stream language program.
3. The method of claim 1 , further comprising:
sending the data of the result stream for display on a screen.
4. The method of claim 1 , wherein one or more data streams from the plurality of data stream are generated as result data streams obtained as a result of execution of instructions of another data stream language program.
5. A computer-readable non-transitory storage medium storing instructions for generating dynamic real-time reports using data streams generated by a data stream language program comprising:
receiving data from a plurality of data streams, each data stream generated by an instance of instrumented software executing on a system;
storing metadata describing the plurality of data streams, the metadata for each data stream including one or more attributes associated with the data stream;
receiving a set of instructions specified using a data stream language program for processing the plurality of data streams, the instructions comprising one or more grouping commands for grouping data streams, each grouping command associated with one or more attributes of data streams and an aggregation operation, and the instructions comprising a publish block, the publish block associated with a metric name used as an attribute;
generating, one or more result data streams by executing the one or more grouping commands to group data derived from the data streams by a set of the one or more attributes of the data streams;
identifying the last grouping command executed in the set of instructions;
determining a set of attributes associated with the last grouping command as a subset of the set of the one or more attributes of the data streams;
for each of the one or more result data streams:
determining a distinct subset of the set of attributes associated with the last grouping command, wherein the distinct subset is used by the last grouping command in generating the result data stream;
determining the values of the attributes in the distinct subset;
concatenating the determined values of the distinct subset;
storing the concatenating values as first metadata describing the result data stream;
storing the metric name of the publish block as a second metadata describing the result data stream;
generating an identifier for the result data stream using the first and second stored metadata describing the result data stream; and
storing the result data stream in association with the identifier;
automatically generating a real-time report identified by the generated identifier and containing the stored result data stream.
6. The computer-readable non-transitory storage medium of claim 5 , further storing instructions for:
using the data of a result stream as input to another block of another data stream language program.
7. The computer-readable non-transitory storage medium of claim 5 , further storing instructions for:
sending the data of a result stream for display on a screen.
8. The computer-readable non-transitory storage medium of claim 5 , wherein one or more data streams from the plurality of data stream are generated as result data streams obtained as a result of execution of instructions of another data stream language program.
9. A computer system comprising: a computer processor; and a computer-readable non-transitory storage medium storing instructions for generating dynamic real-time reports using data streams generated by a data stream language program comprising:
receiving data from a plurality of data streams, each data stream generated by an instance of instrumented software executing on a system;
storing metadata describing the plurality of data streams, the metadata for each data stream including one or more attributes associated with the data stream;
receiving a set of instructions specified using a data stream language program for processing the plurality of data streams, the instructions comprising one or more grouping commands for grouping data streams, each grouping command associated with one or more attributes of data streams and an aggregation operation, and the instructions comprising a publish block, the publish block associated with a metric name used as an attribute;
generating one or more result data streams by executing the one or more grouping commands to group data derived from the data streams by a set of the one or more attributes of the data streams;
identifying the last grouping command executed in the set of instructions;
determining a set of attributes associated with the last grouping command as a subset of the set of the one or more attributes of the data streams;
for each of the one or more result data streams;
determining a distinct subset of the set of attributes associated with the last grouping command, wherein the distinct subset is used by the last grouping command in generating the result data stream;
determining the values of the attributes in the distinct subset;
concatenating the determined values of the distinct subset;
storing the concatenated values as first metadata describing the result data stream;
storing the metric name of the publish block as second metadata describing the result data stream;
generating an identifier for the result data stream using the first and second stored metadata describing the result data stream;
storing the result data stream in association with the identifier; and
automatically generating a real-time report identified by the generated identifier and containing the stored result data stream.
10. The computer system of claim 9 , wherein the computer-readable non-transitory storage medium further comprises stored instructions for:
using the data of a result stream as input to another block of another data stream language program.
11. The computer system of claim 9 , wherein the computer-readable non-transitory storage medium further comprises stored instructions for:
sending the data of a result stream for display on a screen.
12. The computer system of claim 9 , wherein one or more data streams from the plurality of data stream are generated as result data streams obtained as a result of execution of instructions of another data stream language program.