Dynamically changing input data streams processed by data stream language programs
An instrumentation analysis system processes data streams by executing instructions specified using a data stream language program. The data stream language allows users to specify a search condition using a find block for identifying the set of data streams processed by the data stream language program. The set of identified data streams may change dynamically. The data stream language allows users to group data streams into sets of data streams based on distinct values of one or more metadata attributes associated with the input data streams. The data stream language allows users to specify a threshold block for determining whether data values of input data streams are outside boundaries specified using low/high thresholds. The elements of the set of data streams input to the threshold block can dynamically change. The low/high threshold values can be specified as data streams and can dynamically change.
1. A method for processing a data stream language program, the method comprising:
storing, by an instrumentation analysis system, metadata describing a plurality of data streams;
receiving a data stream language program comprising a set of instructions specified using a data stream language, the instructions comprising a find block and a set of blocks following the find block, the find block associated with a search expression, the search expression based on metadata attributes associated with data streams;
repeatedly evaluating the search expression, each evaluation of the search expression identifying a set of data streams from the plurality of data streams conforming to the search expression, wherein the set of identified data streams is provided as input to the set of blocks following the find block; and
repeatedly executing the set of blocks following the find block, the execution comprising:
receiving data values from each data stream of the set of identified data streams,
executing the set of blocks following the find block based on the received data values,
generating one or more result data values based on the execution of the set of blocks, the result values corresponding to result data streams generated by the data stream language program, and
storing the one or more result data values.
2. The method of claim 1 , wherein the set of identified data streams evaluated during a first time interval is distinct from the set of identified data streams evaluated during a second time interval.
3. The method of claim 1 , wherein the find block is associated with a periodicity and the search expression is evaluated periodically for each time interval determined based on a periodicity associated with the find block.
4. The method of claim 1 , wherein the search expression is evaluated in response to a change in metadata associated with the data streams.
5. The method of claim 1 , wherein the search expression is evaluated in response to a change in the plurality of data streams, the change comprising an addition of a data stream or a deletion of a data stream.
6. The method of claim 1 , wherein the search expression is evaluated in response to a change in the specification of the search expression.
7. The method of claim 6 , wherein a rate at which the find block is executed is different from which a rate at which one or more other blocks of the data stream language program are executed.
8. The method of claim 1 , wherein the plurality of data streams comprises data streams received from external systems.
9. The method of claim 1 , wherein the plurality of data streams comprises data streams generated as a result of execution of another data stream language program.
10. The method of claim 1 , wherein the search expression is a regular expression based on metadata associated with data streams.
11. A computer readable non-transitory storage medium storing instructions thereon, the instructions when executed by a processor cause the processor to perform the steps of:
storing, by an instrumentation analysis system, a plurality of data streams, each data stream associated with metadata attributes;
receiving a data stream language program comprising a set of instructions specified using a data stream language, the instructions comprising a find block and a set of blocks following the find block, the find block associated with a search expression, the search expression based on metadata attributes associated with data streams;
repeatedly evaluating the search expression, each evaluation of the search expression identifying a set of data streams from the plurality of data streams conforming to the search expression, wherein the set of identified data streams is provided as input to the set of blocks following the find block; and
repeatedly executing the set of blocks following the find block, the execution comprising:
receiving data values from each data stream of the set of identified data streams,
executing the set of blocks following the find block based on the received data values,
generating one or more result data values based on the execution of the set of blocks, the result values corresponding to result data streams generated by the data stream language program, and
storing the one or more result data values.
12. The computer readable non-transitory storage medium of claim 11 , wherein the set of identified data streams evaluated during a first time interval is distinct from the set of identified data streams evaluated during a second time interval.
13. The computer readable non-transitory storage medium of claim 11 , wherein the find block is associated with a periodicity and the search expression is evaluated periodically for each time interval determined based on a periodicity associated with the find block.
14. The computer readable non-transitory storage medium of claim 11 , wherein the search expression is evaluated in response to a change in metadata associated with the data streams.
15. The computer readable non-transitory storage medium of claim 11 , wherein the search expression is evaluated in response to a change in the plurality of data streams, the change comprising an addition of a data stream or a deletion of a data stream.
16. The computer readable non-transitory storage medium of claim 11 , wherein the find block is associated with a periodicity and the search expression is evaluated periodically for each time interval determined based on a periodicity associated with the find block.
17. The computer readable non-transitory storage medium of claim 16 , wherein a rate at which the find block is executed is different from which a rate at which one or more other blocks of the data stream language program are executed.
18. A computer-implemented system for processing a data stream language program, the system comprising:
a computer processor; and
a computer readable non-transitory storage medium storing instructions thereon, the instructions when executed by a processor cause the processor to perform the steps of:
storing, by an instrumentation analysis system, a plurality of data streams, each data stream associated with metadata attributes;
receiving a data stream language program comprising a set of instructions specified using a data stream language, the instructions comprising a find block and a set of blocks following the find block, the find block associated with a search expression, the search expression based on metadata attributes associated with data streams;
repeatedly evaluating the search expression, each evaluation of the search expression identifying a set of data streams from the plurality of data streams conforming to the search expression, wherein the set of identified data streams is provided as input to the set of blocks following the find block; and
repeatedly executing the set of blocks following the find block, the execution comprising:
receiving data values from each data stream of the set of identified data streams,
executing the set of blocks following the find block based on the received data values,
generating one or more result data values based on the execution of the set of blocks, the result values corresponding to result data streams generated by the data stream language program, and
storing the one or more result data values.
19. The computer-implemented system of claim 18 , wherein the set of identified data streams evaluated during a first time interval is distinct from the set of identified data streams evaluated during a second time interval.
20. The computer-implemented system of claim 18 , wherein the find block is associated with a periodicity and the search expression is evaluated periodically for each time interval determined based on a periodicity associated with the find block.
21. The computer-implemented system of claim 18 , wherein the search expression is evaluated in response to a change in metadata associated with the data streams.
22. The computer-implemented system of claim 18 , wherein the search expression is evaluated in response to a change in the plurality of data streams, the change comprising an addition of a data stream or a deletion of a data stream.
23. The computer-implemented system of claim 18 , wherein the search expression is evaluated in response to a change in the specification of the search expression.
24. The computer-implemented system of claim 23 , wherein a rate at which the find block is executed is different from which a rate at which one or more other blocks of the data stream language program are executed.