Representing result data streams based on execution of data stream language programs
An instrumentation analysis system processes data streams by executing instructions specified using a data stream language program. The data stream language allows users to specify a search condition using a find block for identifying the set of data streams processed by the data stream language program. The set of identified data streams may change dynamically. The data stream language allows users to group data streams into sets of data streams based on distinct values of one or more metadata attributes associated with the input data streams. The data stream language allows users to specify a threshold block for determining whether data values of input data streams are outside boundaries specified using low/high thresholds. The elements of the set of data streams input to the threshold block can dynamically change. The low/high threshold values can be specified as data streams and can dynamically change.
1. A computer-implemented method for processing data streams generated by a data stream language program, the method comprising:
automatically re-evaluating at least one regular expression in a search of a time series data store, wherein the re-evaluating comprises:
receiving data from a plurality of input data streams comprising metric data streams and event data streams, the metric data streams comprising data representing values of metrics of instrumented software, the event data streams comprising conditions and/or actions of external systems;
identifying a subset of the plurality of input data streams having metadata satisfying the regular expression;
generating an intermediate result data stream from the subset of input data streams;
executing a set of instructions comprising a data stream language program to determine a subset of values of an attribute associated with the intermediate result data stream;
while executing the set of instructions of the data stream language program to determine the subset of values, further performing the steps of:
storing the subset of values as metadata describing the intermediate result data stream in time series data store;
generating an identifier for the intermediate result data stream, the identifier comprising a time series identifier;
associating and storing the identifier with the metadata describing the intermediate result data stream in the time series data store;
receiving modifications to the metadata describing the intermediate result data stream;
applying the modifications to the metadata describing the intermediate result data stream;
based on the modifications, dynamically identifying a second subset of the plurality of input data streams having metadata satisfying the regular expression based on the modified metadata describing the intermediate result data stream;
generating a modified first intermediate result data stream from the second subset of input data streams; and
repeating the executing a set of instructions for any further modification to the metadata of the results data stream;
displaying a final modified result data stream via a user interface by a user interface manager, wherein the final modified results data stream is the last first intermediate result data stream based on the modifications to the metadata describing the intermediate result data stream in the time series data store, the user interface is further configured to receive selections of metrics for generating reports, performing rollups, and/or grouping the final modified result data stream; and
displaying aggregates computed over various data centers, different types of applications, different types of servers, or different geographical regions through the user interface manager.
2. The computer-implemented method of claim 1 , further comprising: receiving a dynamic expression for finding a set of result data streams based on an attribute value of the dynamic expression.
3. The computer-implemented method of claim 1 , further comprising: identifying a specific input data stream and a specific result data stream based on a dynamic expression, wherein the specific result data stream is described by an attribute value of the dynamic expression.
4. The computer-implemented method of claim 1 , further comprising: generating an output based on data values of at least one input data stream and at least one result data stream; and providing for display the output through a user interface.
5. The computer-implemented method of claim 1 , further comprising: storing metadata describing the plurality of input data streams, the metadata for each data stream including one or more attributes associated with the data stream.
6. The computer-implemented method of claim 1 , further comprising: receiving a set of instructions specified using a data stream language program for processing the plurality of input data streams; and generating one or more result data streams by executing the set of instructions.
7. The computer-implemented method of claim 1 , wherein a set of result data streams is associated with a publish block, the publish block associated with a metric name, further comprising: determining a metric name attribute by associating the metric name of the publish block with an attribute describing each of the result data streams in the set of result data streams; and storing the metric name attribute with the metadata describing each of the result data streams in the set of result data streams.
8. The computer-implemented method of claim 1 , wherein one or more input data streams from the plurality of input data streams are generated by instances of instrumented software executing on an external system.
9. The computer-implemented method of claim 1 , wherein one or more result data streams are obtained as a result of execution of instructions of another data stream language program.
10. A system for processing data streams generated by a data stream language program, the system comprising:
at least one memory having instructions stored thereon; and
at least one processor configured to execute the instructions, wherein the at least one processor is configured to:
automatically re-evaluate at least one regular expression in a search of a time series data store, wherein re-evaluating comprises:
receiving data from a plurality of input data streams comprising metric data streams and event data streams, the metric data streams comprising data representing values of metrics of instrumented software, the event data streams comprising conditions and/or actions of external systems;
identifying a subset of the plurality of input data streams having metadata satisfying the regular expression;
generating an intermediate result data stream from the subset of input data streams;
executing a set of instructions comprising a data stream language program to determine a subset of values of an attribute associated with the intermediate result data stream;
while executing the set of instructions of the data stream language program to determine the subset of values, further perform the steps of:
storing the subset of values as metadata describing the intermediate result data stream in time series data store;
generating an identifier for the intermediate result data stream, the identifier comprising a time series identifier;
associating and storing the identifier with the metadata describing the intermediate result data stream in the time series data store;
receiving modifications to the metadata describing the intermediate result data stream;
applying the modifications to the metadata describing the intermediate result data stream;
based on the modifications, dynamically identifying a second subset of the plurality of input data streams having metadata satisfying the regular expression based on the modified metadata describing the intermediate result data stream;
generating a modified first intermediate result data stream from the second subset of input data streams; and
repeating the executing a set of instructions for any further modification to the metadata of the results data stream;
displaying a final modified result data stream via a user interface by a user interface manager, wherein the final modified results data stream is the last first intermediate result data stream based on the modifications to the metadata describing the intermediate result data stream in the time series data store, the user interface is further configured to receive selections of metrics for generating reports, performing rollups, and/or grouping the final modified result data stream; and
displaying aggregates computed over various data centers, different types of applications, different types of servers, or different geographical regions through the user interface manager.
11. The system of claim 10 , further configured to: receive a dynamic expression for finding a set of result data streams based on an attribute value of the dynamic expression.
12. The system of claim 10 , further configured to: identify a specific input data stream and a specific result data stream based on a dynamic expression, wherein the specific result data stream is described by an attribute value of the dynamic expression.
13. The system of claim 10 , further configured to: generate an output based on data values of at least one input data stream and at least one result data stream; and provide for display the output through a user interface.
14. The system of claim 10 , further configured to: store metadata describing the plurality of input data streams, the metadata for each data stream including one or more attributes associated with the data stream.
15. The system of claim 10 , further configured to: receive a set of instructions specified using a data stream language program for processing the plurality of input data streams; and generate one or more result data streams by executing the set of instructions.
16. The system of claim 10 , wherein a set of result data streams is associated with a publish block, the publish block associated with a metric name, further configured to: determine a metric name attribute by associating the metric name of the publish block with an attribute describing each of the result data streams in the set of result data streams; and store the metric name attribute with the metadata describing each of the result data streams in the set of result data streams.
17. The system of claim 10 , wherein one or more input data streams from the plurality of input data streams are generated by instances of instrumented software executing on an external system.
18. The system of claim 10 , wherein one or more result data streams are obtained as a result of execution of instructions of another data stream language program.
19. A non-transitory computer-readable storage medium comprising instructions stored thereon, which when executed by one or more processors, cause the one or more processors to perform operations for processing data streams generated by a data stream language program, comprising:
automatically re-evaluating at least one regular expression in a search of a time series data store, wherein the re-evaluating comprises:
receiving data from a plurality of input data streams comprising metric data streams and event data streams, the metric data streams comprising data representing values of metrics of instrumented software, the event data streams comprising conditions and/or actions of external systems;
identifying a subset of the plurality of input data streams having metadata satisfying the regular expression;
generating an intermediate result data stream from the subset of input data streams;
executing a set of instructions comprising a data stream language program to determine a subset of values of an attribute associated with the intermediate result data stream;
while executing the set of instructions of the data stream language program to determine the subset of values, further performing the steps of:
storing the subset of values as metadata describing the intermediate result data stream in time series data store;
generating an identifier for the intermediate result data stream, the identifier comprising a time series identifier;
associating and storing the identifier with the metadata describing the intermediate result data stream in the time series data store;
receiving modifications to the metadata describing the intermediate result data stream;
applying the modifications to the metadata describing the intermediate result data stream;
based on the modifications, dynamically identifying a second subset of the plurality of input data streams having metadata satisfying the regular expression based on the modified metadata describing the intermediate result data stream;
generating a modified first intermediate result data stream from the second subset of input data streams; and
repeating the executing a set of instructions for any further modification to the metadata of the results data stream;
displaying a final modified result data stream via a user interface by a user interface manager, wherein the final modified results data stream is the last first intermediate result data stream based on the modifications to the metadata describing the intermediate result data stream in the time series data store, the user interface is further configured to receive selections of metrics for generating reports, performing rollups, and/or grouping the final modified result data stream; and
displaying aggregates computed over various data centers, different types of applications, different types of servers, or different geographical regions through the user interface manager.
20. The non-transitory computer-readable storage medium of claim 19 , further configured for: receiving a dynamic expression for finding a set of result data streams based on an attribute value of the dynamic expression.
21. The non-transitory computer-readable storage medium of claim 19 , further configured for: identifying a specific input data stream and a specific result data stream based on a dynamic expression, wherein the specific result data stream is described by an attribute value of the dynamic expression.
22. The non-transitory computer-readable storage medium of claim 19 , further configured for: generating an output based on data values of at least one input data stream and at least one result data stream; and providing for display the output through a user interface.
23. The non-transitory computer-readable storage medium of claim 19 , further configured for: storing metadata describing the plurality of input data streams, the metadata for each data stream including one or more attributes associated with the data stream.
24. The non-transitory computer-readable storage medium of claim 19 , further configured for: receiving a set of instructions specified using a data stream language program for processing the plurality of input data streams; and generating one or more result data streams by executing the set of instructions.
25. The non-transitory computer-readable storage medium of claim 19 , wherein a set of result data streams is associated with a publish block, the publish block associated with a metric name, further configured for: determining a metric name attribute by associating the metric name of the publish block with an attribute describing each of the result data streams in the set of result data streams; and storing the metric name attribute with the metadata describing each of the result data streams in the set of result data streams.
26. The non-transitory computer-readable storage medium of claim 19 , wherein one or more input data streams from the plurality of input data streams are generated by instances of instrumented software executing on an external system.
27. The non-transitory computer-readable storage medium of claim 19 , wherein one or more result data streams are obtained as a result of execution of instructions of another data stream language program.