Real-time log analysis service for integrating external event data with log data for use in root cause analysis
A real-time log analyzer (“RTLA”) network service obtains log data from host computers. The RTLA network service also obtains event data from external network services. A user interface (“UI”) associated with the RTLA network service retrieves the log data and the event data from the RTLA network service for a time period, and presents the log data and the event data for the time period together for use in analyzing the root cause of failures indicated in the log data. The UI can also provide functionality for defining the time period, selecting the hosts for which log data and event data is to be presented, specifying the type of log data and event data to be displayed or otherwise filtering the log and event data, displaying data from a trouble ticket system, defining and displaying annotations relating to the event data, and/or displaying a human-friendly description of the event data.
1. An apparatus, comprising:
one or more processors; and
one or more non-transitory computer-readable storage media having instructions stored thereupon which are executable by the one or more processors and which, when executed, cause the apparatus to:
receive a request from a computing device for log data for a time period, the log data being generated by a plurality of host computers;
responsive to the request for the log data, obtain the log data for the time period and provide the log data to the computing device;
receive a request from the computing device for event data for the time period, the event data being generated by one or more network services;
responsive to the request for the event data, query a distributed cache for the event data for the time period;
responsive to the query locating the event data for the time period in the distributed cache, retrieve the event data from the distributed cache and return the event data for the time period to the computing device in response to the request; and
responsive to the query not locating the event data for the time period in the distributed cache,
obtain the event data for the time period from the one or more network services,
store the event data for the time period obtained from the one or more network services in the distributed cache, and
return the event data for the time period to the computing device in response to the request,
wherein the computing device is configured to present a user interface (UI) comprising the log data for the time period and the event data for the time period.
2. The apparatus of claim 1 , wherein the one or more non-transitory computer-readable storage media have further instructions stored thereupon to:
set a time-to-live (TTL) value for the event data in the distributed cache, the TTL value comprising a first value if the time period is in the past and comprising a second value if the time period is not in the past.
3. The apparatus of claim 1 , wherein the UI is further configured to present one or more UI controls for specifying the time period.
4. The apparatus of claim 1 , wherein the UI is further configured to present one or more UI controls for specifying the plurality of host computers.
5. The apparatus of claim 1 , wherein the UI is further configured to present one or more UI controls for specifying a type of the event data.
6. The apparatus of claim 1 , wherein the UI is further configured to present one or more UI controls for displaying data obtained from a trouble ticket system relating to the event data.
7. The apparatus of claim 1 , wherein the UI is further configured to present one or more UI controls for defining and displaying an annotation relating to the event data.
8. A computer-implemented method, comprising:
causing one or more graphical user interface (UI) controls to be displayed in a first region of an interactive UI, the one or more UI controls enabling selection of a plurality of host computers and a period of time;
causing a graphical pane to be displayed in a second region of the interactive UI;
receiving a selection of the plurality of host computers and the period of time on the one or more UI controls;
retrieving log data for the plurality of host computers for the time period from a network service;
retrieving event data for the time period from the network service, the event data describing events related to operation of the plurality of host computers and occurring at one or more network services external to the plurality of host computers, wherein retrieving event data for the time period from the network service comprises:
querying a distributed cache for the event data for the time period;
in response to the querying locating the event data for the time period in the distributed cache, retrieving the event data from the distributed cache; and
in response to the querying not locating the event data for the time period in the distributed cache,
obtaining the event data for the time period from the one or more network services, and
storing the event data for the time period obtained from the one or more network services in the distributed cache; and
causing the graphical pane of the interactive UI to be updated to present the event data graphically integrated with the log data for the time period.
9. The computer-implemented method of claim 8 , wherein the interactive UI is further configured to present one or more UI controls for specifying a type of the event data.
10. The computer-implemented method of claim 8 , wherein the interactive UI is further configured to present one or more UI controls for displaying data obtained from a trouble ticket system relating to the event data.
11. The computer-implemented method of claim 8 , wherein the interactive UI is further configured to present one or more UI controls for defining and displaying an annotation relating to the event data.
12. The computer-implemented method of claim 8 , wherein the interactive UI is further configured to present a text description of the event data.
13. A non-transitory computer-readable storage media having instructions stored thereupon that are executable by one or more processors and which, when executed, cause the one or more processors to:
cause one or more graphical user interface (UI) controls to be displayed in a first region of an interactive UI, the one or more UI controls enabling selection of a plurality of host computers and a period of time;
cause a graphical pane to be displayed in a second region of the interactive UI;
receive a selection of the plurality of host computers and the period of time on the one or more UI controls;
retrieve log data for a plurality of host computers for a time period from a network service;
retrieve event data for the time period from the network service, the event data describing events related to operation of the plurality of host computers and occurring at one or more network services external to the plurality of host computers, wherein retrieving event data for the time period from the network service comprises to:
query a distributed cache for the event data for the time period;
in response to the query locating the event data for the time period in the distributed cache, retrieve the event data from the distributed cache; and
in response to the query not locating the event data for the time period in the distributed cache,
obtain the event data for the time period from the one or more network services, and
store the event data for the time period obtained from the one or more network services in the distributed cache; and
cause the graphical pane of the interactive UI to be updated to present the event data graphically integrated with the log data for the time period.
14. The non-transitory computer-readable storage media of claim 13 , wherein the interactive UI is further configured to present one or more UI controls for displaying data obtained from a trouble ticket system relating to the event data.
15. The non-transitory computer-readable storage media of claim 13 , wherein the interactive UI is further configured to present one or more UI controls for defining and displaying an annotation relating to the event data.
16. The non-transitory computer-readable storage media of claim 13 , wherein the interactive UI is further configured to present a text description of the event data.
17. The computer-implemented method of claim 8 , wherein the event data describes deployments of program code to the plurality of host computers by a deployment service external to the plurality of host computers.
18. The non-transitory computer-readable storage media of claim 13 , wherein the event data describes deployments of program code to the plurality of host computers by a deployment service external to the plurality of host computers.
19. The computer-implemented method of claim 8 , further comprising:
setting a time-to-live (TTL) value for the event data in the distributed cache, the TTL value comprising a first value if the time period is in the past and comprising a second value if the time period is not in the past.
20. The non-transitory computer-readable storage media of claim 13 , wherein the instructions are further executable by the one or more processors to:
set a time-to-live (TTL) value for the event data in the distributed cache, the TTL value comprising a first value if the time period is in the past and comprising a second value if the time period is not in the past.