Query-triggered processing of performance data and log data from an information technology environment
The disclosed system and method acquire and store performance measurements relating to performance of a component in an information technology (IT) environment and log data produced by the IT environment, in association with corresponding time stamps. The disclosed system and method correlate at least one of the performance measurements with at least one of the portions of log data.
1. A method comprising:
acquiring, by a computer system, a plurality of performance measurements for a performance metric associated with at least one hardware or software component of an information technology (IT) environment;
acquiring, by the computer system, a plurality of portions of log data representing activity of the at least one hardware or software component of the IT environment;
storing, by the computer system, the acquired performance measurements in a first time-series data store in a first format and the acquired portions of log data in a second time-series data store in a second format different from the first format, the second time-series data store being separate from the first time-series data store;
inputting, by the computer system, a search query expression of a first search criterion comprising a command and a corresponding argument and a second search criterion, the search query expression specified by a user in a text-based search query language via a single query box, wherein the first search criterion is not a time-based criterion and the second search criterion is a time-based criterion; and
correlating, by the computer system, at least one of the performance measurements with at least one of the portions of log data for a selected hardware or software component of the IT environment, based on the first search criterion and the second search criterion, wherein said correlating includes:
in response to the user-specified search query expression in the text-based search query language input via the single query box, applying the user-specified search query expression in the text-based search query language to the performance measurements stored in the first time-series data store in the first format and the portions of log data stored in the second time-series data store in the second format different from the first format,
causing display of an indication that the at least one performance measurement satisfies the first search criterion and the second search criterion, and
causing display of an indication that the at least one portion of log data satisfies the first search criterion and the second search criterion.
2. A method as recited in claim 1 , wherein the selected hardware or software component of the IT environment is user-selected via a displayed architecture defining a structure of components of the IT environment and relationships thereof.
3. A method as recited in claim 1 , wherein storing comprises:
storing the acquired performance measurements as time-stamped performance events and storing the acquired portions of log data as time-stamped log events, such that each performance event and each log event is stored in association with a respective time stamp.
4. A method as recited in claim 1 , wherein:
storing comprises storing the acquired performance measurements as time-stamped performance events and storing the acquired portions of log data as time-stamped log events, such that each performance event and each log event is stored in association with a respective time stamp; and
correlating further comprises identifying at least one of the stored performance events and at least one of the stored log events whose time stamps each satisfy the second search criterion comprising the time-based criterion.
5. A method as recited in claim 1 , further comprising:
acquiring, by the computer system, a plurality of performance measurements for each of a plurality of performance metrics, the plurality of performance metrics associated with a plurality of components of the IT environment.
6. A method as recited in claim 1 , wherein correlating further comprises causing concurrent display of the performance measurements that satisfy the first search criterion and second search criterion and a listing of raw log data that satisfy the first search criterion and the second search criterion.
7. A method as recited in claim 1 , further comprising:
obtaining a third search criterion by a user input to a drop-down list in a graphical user interface;
wherein correlating further comprises identifying a performance measurement and a portion of log data that both satisfy the third search criterion.
8. A method as recited in claim 1 , further comprising:
obtaining a third search criterion by a user input to a drop-down list in a graphical user interface, wherein the third search criterion relates to a machine in the IT environment;
wherein correlating further comprises identifying a performance measurement and a portion of log data that both satisfy the third search criterion.
9. A method as recited in claim 1 , wherein correlating comprises identifying at least one of the stored performance measurements and at least one of the stored portions of log data that have time stamps that satisfy the second search criterion.
10. A method as recited in claim 1 , wherein the performance measurements have been determined by direct measurement of a hardware or software component in the IT environment.
11. A method as recited in claim 1 , wherein the plurality of portions of log data are from a text-based log file.
12. A method as recited in claim 1 , wherein the plurality of performance measurements is not derived from a log file and are acquired independently of the plurality of portions of log data.
13. A method as recited in claim 1 , wherein the plurality of performance measurements is acquired independently of the plurality of portions of log data, and wherein the performance measurements have been determined by direct measurement of a hardware or software component in the IT environment and the plurality of portions of log data are from a text-based log file.
14. A method as recited in claim 1 , wherein the plurality of performance measurements is acquired independently of the plurality of portions of log data by direct measurement of a hardware or software component in the IT environment, and the plurality of portions of log data are acquired independently of the performance measurements.
15. A method as recited in claim 1 , wherein acquiring the plurality of performance measurements comprises acquiring, via an application programming interface (API), the plurality of performance measurements from a third-party software application that collects the performance measurements.
16. A method as recited in claim 1 , further comprising:
acquiring structure data indicative of structure characteristics of the IT environment;
storing the acquired structure data indicative of structure characteristics of the IT environment; and
correlating a performance characteristic of the IT environment with a structure characteristic of the IT environment, based on the stored performance measurements and stored structure data.
17. A method as recited in claim 1 , further comprising:
acquiring structure data indicative of structure characteristics of the IT environment, wherein the structure data is derived from log data from the IT environment;
storing the acquired structure data indicative of structure characteristics of the IT environment; and
correlating a performance characteristic of the IT environment with a structure characteristic of the IT environment, based on the stored performance measurements and stored structure data.
18. A method as recited in claim 1 , wherein the at least one performance measurement is of a virtual machine host in the IT environment, and wherein the at least one portion of log data is of the virtual machine host.
19. A method as recited in claim 1 , wherein the at least one performance measurement is of a cluster of virtual machine hosts in the IT environment, and wherein the at least one portion of log data is of one of the virtual machine hosts.
20. A method as recited in claim 1 , wherein the at least one performance measurement is of a virtual machine in the IT environment, and wherein the at least one portion of log data is of a host of the virtual machine.
21. A method as recited in claim 1 , wherein the first search criterion is a user-specified value or a user-specified range of values, for a user-specified field.
22. A method as recited in claim 1 , wherein:
said correlating includes, in response to the search query expression, searching the first time-series data store for performance data that satisfy the first search criterion and the second search criterion, and searching the second time-series data store for log data that satisfy the first search criterion and the second search criterion.
23. A method as recited in claim 1 , wherein:
the first search criterion comprises a user-specified value or a user-specified range of values, for a user-specified field; and
said correlating includes, in response to the search query expression, searching the first time-series data store for performance data that satisfy the first search criterion and the second search criterion, and searching the second time-series data store for log data that satisfy the first search criterion and the second search criterion.
24. A non-transitory machine-readable storage medium for use in a processing system of a data intake and query system, the non-transitory machine-readable storage medium storing instructions, an execution of which in the processing system causes the processing system to perform operations comprising:
acquiring, by a computer system, a plurality of performance measurements for a performance metric associated with at least one hardware or software component of an information technology (IT) environment;
acquiring, by the computer system, a plurality of portions of log data representing activity of the at least one hardware or software component of the IT environment;
storing, by the computer system, the acquired performance measurements in a first time-series data store in a first format and the acquired portions of log data in a second time-series data store in a second format different from the first format, the second time-series data store being separate from the first time-series data store;
inputting, by the computer system, a search query expression of a first search criterion comprising a command and a corresponding argument and a second search criterion, the search query expression specified by a user in a text-based search query language via a single query box, wherein the first search criterion is not a time-based criterion and the second search criterion is a time-based criterion; and
correlating, by the computer system, at least one of the performance measurements with at least one of the portions of log data for a selected hardware or software component of the IT environment, based on the first search criterion and the second search criterion, wherein said correlating includes:
in response to the user-specified search query expression in the text-based search query language input via the single query box, applying the user-specified search query expression in the text-based search query language to the performance measurements stored in the first time-series data store in the first format and the portions of log data stored in the second time-series data store in the second format different from the first format,
causing display of an indication that the at least one performance measurement satisfies the first search criterion and the second search criterion, and
causing display of an indication that the at least one portion of log data satisfies the first search criterion and the second search criterion.
25. A system comprising:
a communication device through which to communicate on a computer network; and
at least one processor operatively coupled to the communication device and configured to perform operations including:
acquiring, by a computer system, a plurality of performance measurements for a performance metric associated with at least one hardware or software component of an information technology (IT) environment;
acquiring, by the computer system, a plurality of portions of log data representing activity of the at least one hardware or software component of the IT environment;
storing, by the computer system, the acquired performance measurements in a first time-series data store in a first format and the acquired portions of log data in a second time-series data store in a second format different from the first format, the second time-series data store being separate from the first time-series data store;
inputting, by the computer system, a search query expression of a first search criterion comprising a command and a corresponding argument and a second search criterion, the search query expression specified by a user in a text-based search query language via a single query box, wherein the first search criterion is not a time-based criterion and the second search criterion is a time-based criterion; and
correlating, by the computer system, at least one of the performance measurements with at least one of the portions of log data for a selected hardware or software component of the IT environment, based on the first search criterion and the second search criterion, wherein said correlating includes:
in response to the user-specified search query expression in the text-based search query language input via the single query box, applying the user-specified search query expression in the text-based search query language to the performance measurements stored in the first time-series data store in the first format and the portions of log data stored in the second time-series data store in the second format different from the first format,
causing display of an indication that the at least one performance measurement satisfies the first search criterion and the second search criterion, and
causing display of an indication that the at least one portion of log data satisfies the first search criterion and the second search criterion.