Dynamically changing input data streams processed by data stream language programs
An instrumentation analysis system processes data streams by executing instructions specified using a data stream language program. The data stream language allows users to specify a search condition using a find block for identifying the set of data streams processed by the data stream language program. The set of identified data streams may change dynamically. The data stream language allows users to group data streams into sets of data streams based on distinct values of one or more metadata attributes associated with the input data streams. The data stream language allows users to specify a threshold block for determining whether data values of input data streams are outside boundaries specified using low/high thresholds. The elements of the set of data streams input to the threshold block can dynamically change. The low/high threshold values can be specified as data streams and can dynamically change.
1. A method for generating and storing data streams generated by a data stream language program, the method comprising:
receiving data from a plurality of data streams, each data stream received from an instance of instrumented software executing on an external system;
storing metadata describing the plurality of data streams, the metadata for each data stream including one or more attributes associated with the data stream;
receiving a set of instructions specified using a data stream language program for processing the plurality of data streams;
generating one or more result data streams by executing the set of instructions;
for each of the one or more result data streams:
determining a set of values of attributes describing the result data stream, the determining based on the set of instructions;
storing the set of values as metadata describing the result data stream;
generating an identifier for tuples of the result data stream and associating the identifier with the metadata describing the tuples of the result data stream, each tuple comprising at least a metric name; and
storing data of the result data stream in association with the identifier; and
providing one or more result data streams as input to another data stream language program.
2. The method of claim 1 , wherein the data stream language program comprises a grouping command specifying grouping of data across data streams, wherein the grouping command identifies one or more attributes and wherein the set of values corresponds to the one or more attributes of the grouping command specified in the data stream language program.
3. The method of claim 2 , wherein the grouping command groups data of the data streams by a set of metadata attributes and each result data stream corresponds to a distinct set of values of the set of metadata attributes.
4. The method of claim 1 , wherein the data stream language program specifies a plurality of grouping commands, each of the plurality of grouping commands identifying one or more attributes, and wherein the set of values is based on the one or more attributes of the last grouping command from the plurality of grouping commands specified in the data stream language program.
5. The method of claim 1 , further comprising:
sending the data of the result stream for display on a screen.
6. The method of claim 1 , wherein the set of result data streams is associated with a publish block, the publish block associated with the metric name, further comprising:
for each result data stream:
using the metric name of the publish block as an attribute describing the result data stream; and
storing the attribute based on the metric name with metadata describing the result data stream.
7. The method of claim 1 , wherein one or more data streams from the plurality of data streams are generated as result data streams obtained as a result of execution of instructions of another data stream language program.
8. The method of claim 1 , wherein the metadata describing the plurality of data streams is received independent of the data of the data stream.
9. A computer-readable non-transitory storage medium storing instructions for:
receiving data from a plurality of data streams, each data stream received from an instance of instrumented software executing on an external system;
storing metadata describing the plurality of data streams, the metadata for each data stream including one or more attributes associated with the data stream;
receiving a set of instructions specified using a data stream language program for processing the plurality of data streams;
generating one or more result data streams by executing the set of instructions;
for each of the one or more result data streams:
determining a set of values of attributes describing the result data stream, the determining based on the set of instructions;
storing the set of values as metadata describing the result data stream;
generating an identifier for tuples of the result data stream and associating the identifier with the metadata describing the tuples of the result data stream, each tuple comprising at least a metric name; and
storing data of the result data stream in association with the identifier; and
providing one or more result data streams as input to another data stream language program.
10. The computer-readable non-transitory storage medium of claim 9 , wherein the data stream language program comprising a grouping command specifying grouping of data across data streams, wherein the grouping command identifies one or more attributes and wherein the set of values corresponds to the one or more attributes of the grouping command specified in the data stream language program.
11. The computer-readable non-transitory storage medium of claim 10 , wherein the grouping command groups data of the data streams by a set of metadata attributes and each result data stream corresponds to a distinct set of values of the set of metadata attributes.
12. The computer-readable non-transitory storage medium of claim 9 , wherein the data stream language program specifies a plurality of grouping commands, each of the plurality of grouping commands identifying one or more attributes, and wherein the set of values is based on the one or more attributes of the last grouping command from the plurality of grouping commands specified in the data stream language program.
13. The computer-readable non-transitory storage medium of claim 9 , wherein the set of result data streams is associated with a publish block, the publish block associated with the metric name, further storing instructions for:
for each result data stream:
using the metric name of the publish block as an attribute describing the result data stream; and
storing the attribute based on the metric name with metadata describing the result data stream.
14. The computer-readable non-transitory storage medium of claim 9 , further storing instructions for:
providing one or more result data streams as input to another data stream language program.
15. The computer-readable non-transitory storage medium of claim 9 , further storing instructions for:
sending data of a result stream for display on a screen.
16. The computer-readable non-transitory storage medium of claim 9 , wherein one or more data streams from the plurality of data streams are generated as result data streams obtained as a result of execution of instructions of another data stream language program.
17. A computer system comprising:
a computer processor; and
a computer-readable non-transitory storage medium storing instructions for:
receiving data from a plurality of data streams, each data stream received from an instance of instrumented software executing on an external system;
storing metadata describing the plurality of data streams, the metadata for each data stream including one or more attributes associated with the data stream;
receiving a set of instructions specified using a data stream language program for processing the plurality of data streams;
generating one or more result data streams by executing the set of instructions;
for each of the one or more result data streams:
determining a set of values of attributes describing the result data stream, the determining based on the set of instructions;
storing the set of values as metadata describing the result data stream;
generating an identifier for tuples of the result data stream and associating the identifier with the metadata describing the tuples of the result data stream, each tuple comprising at least a metric name; and
storing data of the result data stream in association with the identifier; and
providing one or more result data streams as input to another data stream language program.
18. The computer system of claim 17 , wherein the data stream language program comprising a grouping command specifying grouping of data across data streams, wherein the grouping command identifies one or more attributes and wherein the set of values corresponds to the one or more attributes of the grouping command specified in the data stream language program.
19. The computer system of claim 18 , wherein the grouping command groups data of the data streams by a set of metadata attributes and each result data stream corresponds to a distinct set of values of the set of metadata attributes.
20. The computer system of claim 17 , wherein the data stream language program specifies a plurality of grouping commands, each of the plurality of grouping commands identifying one or more attributes, and wherein the set of values is based on the one or more attributes of the last grouping command from the plurality of grouping commands specified in the data stream language program.