Associating metadata with results produced by applying a pipelined search command to machine data in timestamped events
View Patent ↗Embodiments are directed towards determining and tracking metadata for the generation of visualizations of requested data. A user may request data by providing a query that may be employed to search for the requested data. The query may include a plurality of commands, which may be employed in a pipeline to perform the search and to generate a table of the requested data. In some embodiments, each command may be executed to perform an action on a set of data. The execution of a command may generate one or more columns to append and/or insert into the table of requested data. Metadata for each generated column may be determined based on the actions performed by executing the commands. The table of requested data and the column metadata may be employed to generate and display a visualization of at least a portion of the requested data to a user.
1. A method in a computing system for performing a pipelined search, comprising:
(a) receiving a search query specifying a first processing command and a second processing command, the search query indicating that the first processing command precedes the second processing command;
(b) accessing a plurality of timestamped events derived from machine data;
(c) applying the first processing command to data from the accessed plurality of timestamped events to obtain a first result set;
(d) associating, with the first result set, metadata reflecting the type of data that resulted from applying the first processing command to data from the accessed plurality of timestamped events;
(e) applying the second processing command to data from the first result set to obtain a second result set;
(f) associating with the second result set, metadata reflecting the type of data that resulted from applying the second processing command to data from the first result set; and
(g) causing display of both the first and second result sets based on their associated metadata.
2. The method of claim 1 wherein the received search query further specifies a third processing command, the search query further indicating that the second processing command precedes the third processing command,
the method further comprising:
applying the third processing command to data from the second result set to obtain a third result set.
3. The method of claim 1 wherein the received search query further specifies the accessed plurality of timestamped events.
4. The method of claim 1 wherein each of at least a portion of the accessed plurality of timestamped events identifies a network host that originated machine data to which the timestamped event corresponds.
5. The method of claim 1 wherein (e) begins before (c) is completed.
6. The method of claim 1 wherein (e) begins after (c) is completed.
7. The method of claim 1 wherein the search query indicates that the first processing command precedes the second processing command by comprising a string in which a first substring referring to the first processing command is followed by a delimiter substring, and wherein the delimiter substring is followed by a second substring referring to the second processing command.
8. The method of claim 1 wherein the search query indicates that the first processing command precedes the second processing command by comprising a string in which a first substring referring to the first processing command is followed by a pipe character “|”, and wherein the pipe character is followed by a second substring referring to the second processing command.
9. The method of claim 1 wherein the search query further specifies a condition by which the first processing command is qualified,
the method further comprising:
for each of the accessed plurality of timestamped events:
where the condition is satisfied, processing the timestamped events in accordance with the first processing command; and
where the condition is not satisfied, passing the timestamped event to the first result set in the form in which it occurs in the accessed plurality of timestamped events.
10. The method of claim 1 wherein the search query further specifies a condition by which the second processing command is qualified,
the method further comprising:
for each of the first result set:
where the condition is satisfied, processing the timestamped events in accordance with the first processing command; and
where the condition is not satisfied, passing the timestamped event to the second result set in the form in which it occurs in the first result set.
11. The method of claim 1 wherein at least one of the first and second processing commands comprises an event filtering operation.
12. The method of claim 1 wherein at least one of the first and second processing commands comprises an event transformation operation.
13. The method of claim 1 wherein at least one of the first and second processing commands comprises a data lookup operation.
14. The method of claim 1 wherein at least one of the first and second processing commands comprises a field extraction operation.
15. The method of claim 1 wherein at least one of the first and second processing commands comprises an aggregation operation.
16. The method of claim 1 wherein at least one of the first and second processing commands comprises a calculation operation.
17. The method of claim 1 , further comprising causing information describing the second result set to be displayed in a manner based at least in part on the metadata associated with the second result set.
18. The method of claim 1 , further comprising causing to be displayed contents of at least one event in the second result set.
19. The method of claim 1 , further comprising causing information summarizing the second result set to be displayed in a manner based at least in part on the metadata associated with the second result set.
20. The method of claim 1 , further comprising causing a bar graph summarizing the second result set to be displayed in a manner based at least in part on the metadata associated with the second result set.
21. The method of claim 1 , further comprising causing a line chart summarizing the second result set to be displayed in a manner based at least in part on the metadata associated with the second result set.
22. The method of claim 1 , further comprising causing a pie chart summarizing the second result set to be displayed in a manner based at least in part on the metadata associated with the second result set.
23. The method of claim 1 , further comprising causing a table summarizing the second result set to be displayed in a manner based at least in part on the metadata associated with the second result set.
24. The method of claim 1 , further comprising causing a crosstab table summarizing the second result set to be displayed in a manner based at least in part on the metadata associated with the second result set.
25. The method of claim 1 wherein the second result set obtained by applying the second processing command comprises a plurality of results,
the method further comprising:
for each of the plurality of results of the second result set, determining both a category and a subcategory; and
causing at least a portion of the plurality of results to be displayed in connection with the categories and subcategories determined for them in a manner based at least in part on the metadata associated with the second result set.
26. A non-transitory computer-readable medium having contents configured to cause a computing system to, in order to perform a pipelined search:
receive a search query specifying a first processing command and a second processing command, the search query indicating that the first processing command precedes the second processing command;
access a plurality of timestamped events derived from machine data;
apply the first processing command to data from the accessed plurality of timestamped events to obtain a first result set;
associate, with the first result set, metadata reflecting the type of data that resulted from applying the first processing command to data from the accessed plurality of timestamped events;
apply the second processing commands to data from the first result set to obtain a second result set; and
associate, with the second result, set metadata reflecting the type of data that resulted from applying the second processing command to data from the first result set; and
generate a display of the first and second result sets based on their respective associated metadata.
27. The non-transitory computer-readable medium of claim 26 wherein the search query indicates that the first processing command precedes the second processing command by comprising a string in which a first substring referring to the first processing command is followed by a delimiter substring, and wherein the delimiter substring is followed by a second substring referring to the second processing command.
28. A computing system for performing a pipelined search, comprising:
a receiving subsystem configured to receive a search query specifying a first processing command and a second processing command, the search query indicating that the first processing command precedes the second processing command;
an event accessing subsystem configured to access a plurality of timestamped events derived from machine data;
a processor configured as a processing command application subsystem to:
apply the first processing command to data from the accessed plurality of timestamped events to obtain a first result set, and
applying the second processing command to data from the first result set to obtain a second result set; and
a metadata management subsystem configured to:
associate with the first result set metadata reflecting the type of data that resulted from applying the first processing command to data from the accessed plurality of timestamped events, and
associate with the second result set metadata reflecting the type of data that resulted from applying the second processing command to data from the first result set.
29. The computing system of claim 28 wherein the search query received by the receiving subsystem indicates that the first processing command precedes the second processing command by comprising a string in which a first substring referring to the first processing command is followed by a pipe character “|”, and wherein the pipe character is followed by a second substring referring to the second processing command.
30. The computing system of claim 28 wherein at least one of the first and second processing commands specified by the search query received by the receiving subsystem comprises an event filtering operation.