Combining data streams generated by instrumented software using metadata correlation
A system processes data stream language expressions that combine result data streams from multiple data stream language sub-expressions. The system determines a set of fixed dimensions based on static analysis of the data stream language sub-expression. The system determines a union set representing a union of the sets of fixed dimensions. The system determines at execution time of the data stream language expression, a plurality of sets of data streams. Each set of data stream corresponds to a data stream language sub-expression from the plurality of data stream language expressions. The system correlates data streams across the plurality of sets of data streams based on the union set. The system determines result data streams for the data stream language expression by combining data values of correlated data streams.
1. A computer-implemented method for evaluating expressions and processing data streams, the method comprising:
determining a set of dimensions based at least in part on an analysis of a data stream language sub-expression, the set of dimensions comprising keys that identify data streams obtained by processing the data stream language sub-expression;
determining a union set comprising a union of sets of dimensions corresponding to a plurality of data stream language sub-expressions;
determining a plurality of sets of data streams, each set of the plurality of sets of data streams corresponding to at least one data stream language sub-expression of the plurality of data stream language sub-expressions;
correlating data streams across the plurality of sets of data streams based on the union set;
determining one or more result data streams for the data stream language sub-expression, wherein each data value of a result data stream comprises combined data values of correlated data streams from the plurality of sets of data streams; causing display of the one or more result data streams through a user interface;
identifying a data value from a designated set of data streams; identifying a matching data stream for each remaining set of data streams;
identifying a data value from the matching data stream; and combining all of the identified data values based on the data stream language sub-expression.
2. The computer-implemented method of claim 1 , wherein correlating the data streams across the plurality of sets of data streams comprises:
selecting a designated set of data streams from the plurality of sets of data streams; and
matching data streams from the designated set of data streams with data streams from each remaining set of data streams from the plurality of sets of data streams.
3. The computer-implemented method of claim 1 , wherein a designated set of data streams corresponds to the set of data streams from the plurality of data streams with a most specific grouping of correlating keys.
4. The computer-implemented method of claim 1 , wherein a key based on a first set of dimensions corresponding to a designated set of data streams matches a key based on a second set of dimensions if the second set of dimensions is a subset of the first set of dimensions.
5. The computer-implemented method of claim 1 , further comprising:
storing correlation keys of a designated set of data streams as metadata for a result set of data streams obtained by evaluation of the data stream language sub-expression.
6. The computer-implemented method of claim 1 , further comprising repeatedly evaluating the data stream language sub-expression at a predetermined rate.
7. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform operations for evaluating expressions and processing data streams, including:
determining a set of dimensions based at least in part on an analysis of a data stream language sub-expression, the set of dimensions comprising keys that identify data streams obtained by processing the data stream language sub-expression;
determining a union set comprising a union of sets of dimensions corresponding to a plurality of data stream language sub-expressions;
determining a plurality of sets of data streams, each set of the plurality of sets of data streams corresponding to at least one data stream language sub-expression of the plurality of data stream language sub-expressions;
correlating data streams across the plurality of sets of data streams based on the union set;
determining one or more result data streams for the data stream language sub-expression, wherein each data value of a result data stream comprises combined data values of correlated data streams from the plurality of sets of data streams; causing display of the one or more result data streams through a user interface;
identifying a data value from a designated set of data streams;
identifying a matching data stream for each remaining set of data streams;
identifying a data value from the matching data stream; and combining all of the identified data values based on the data stream language sub-expression.
8. The non-transitory computer-readable medium of claim 7 , wherein the operations for correlating the data streams across the plurality of sets of data streams further include:
selecting a designated set of data streams from the plurality of sets of data streams; and
matching data streams from the designated set of data streams with data streams from each remaining set of data streams from the plurality of sets of data streams.
9. The non-transitory computer-readable medium of claim 7 , wherein a designated set of data streams corresponds to the set of data streams from the plurality of data streams with a most specific grouping of correlating keys.
10. The non-transitory computer-readable medium of claim 7 , wherein a key based on a first set of dimensions corresponding to a designated set of data streams matches a key based on a second set of dimensions if the second set of dimensions is a subset of the first set of dimensions.
11. The non-transitory computer-readable medium of claim 7 , wherein the operations further include:
storing correlation keys of a designated set of data streams as metadata for a result set of data streams obtained by evaluation of the data stream language sub-expression.
12. The non-transitory computer-readable medium of claim 7 , wherein the operations further include repeatedly evaluating the data stream language sub-expression at a predetermined rate.
13. A computing device, comprising: a processor; and a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations for evaluating expressions and processing data streams, including:
determining a set of dimensions based at least in part on an analysis of a data stream language sub-expression, the set of dimensions comprising keys that identify data streams obtained by processing the data stream language sub-expression;
determining a union set comprising a union of sets of dimensions corresponding to a plurality of data stream language sub-expressions;
determining a plurality of sets of data streams, each set of the plurality of sets of data streams corresponding to at least one data stream language sub-expression of the plurality of data stream language sub-expressions;
correlating data streams across the plurality of sets of data streams based on the union set;
determining one or more result data streams for the data stream language sub-expression, wherein each data value of a result data stream comprises combined data values of correlated data streams from the plurality of sets of data streams; causing display of the one or more result data streams through a user interface;
identifying a data value from a designated set of data streams;
identifying a matching data stream for each remaining set of data streams;
identifying a data value from the matching data stream; and combining all of the identified data values based on the data stream language sub-expression.
14. The computing device of claim 13 , wherein the operations for correlating the data streams across the plurality of sets of data streams further include:
selecting a designated set of data streams from the plurality of sets of data streams; and
matching data streams from the designated set of data streams with data streams from each remaining set of data streams from the plurality of sets of data streams.
15. The computing device of claim 13 , wherein a designated set of data streams corresponds to the set of data streams from the plurality of data streams with a most specific grouping of correlating keys.
16. The computing device of claim 13 , wherein a key based on a first set of dimensions corresponding to a designated set of data streams matches a key based on a second set of dimensions if the second set of dimensions is a subset of the first set of dimensions.
17. The computing device of claim 13 , wherein the operations further include:
storing correlation keys of a designated set of data streams as metadata for a result set of data streams obtained by evaluation of the data stream language sub-expression.