Centralized log visualization for analysis debugging in cluster networks
A multi-node, multi-container cluster system that generates, aggregates, and manages log files from services and components to be used for audit logs and to debug and perform other serviceability tasks provided by a vendor of the cluster system. Logs are collected from all components of the system and aggregated into a consistent format for user analysis and debugging. Embodiments provide a comprehensive way to parse and index vast numbers of log files that can then be packaged and displayed to a user in a way that facilitates analysis and debugging and/or efficient input to appropriate debugging programs.
1 . A method of processing logs for centralized log visualization in a cluster system operated by a user and having a plurality of nodes executing containerized applications, the method comprising:
collecting log files containing logs for each node of a plurality of nodes in the cluster system, wherein the logs record transactions of components of the system;
indexing the collected logs by assigning a unique index to each log file to generate indexed log files;
processing each indexed log file through a pipeline to transform unstructured log information into structured log data; and
displaying the structured log data through a graphical user interface (GUI) for visual analysis by a user to debug an issue related to a corresponding log message, wherein the cluster system comprises a Santorini network processing containerized data utilizing a Kubernetes-based framework, and wherein the cluster network comprises part of a Data Domain deduplication backup system performing backup and restore operations for the nodes.
2 . The method of claim 1 wherein the GUI displays a histogram of logged event occurrences over a user-defined period of time.
3 . The method of claim 2 wherein the GUI further displays detailed textual information of each logged event comprising at least date of event, log identifier, log message, and log file path.
4 . The method of claim 1 further comprising sending the structured log data to a debugger process to debug the issue.
5 . The method of claim 4 wherein the debugger process is executed at one of: a resident site of the cluster system operated by the user, or a vendor site providing debugging services.
6 . The method of claim 5 wherein the logs are collected for events including system changes, authorization activities, privileged access events, and audit related activities.
7 . The method of claim 6 wherein the system changes comprise component failures and changes to availability, configuration, or source code; the authorization activities comprise login or access failures and access provisioning; the privileged access events comprise using a superuser status or running an administrator console; and the audit-related activities comprise changing event logging configurations, deleting event logs, or audit log failures.
8 . The method of claim 1 further comprising:
writing logs for critical components to dedicated persistent volumes for each respective component;
writing logs for non-critical components to a central persistent volume (PV); and
combining the collected logs for the critical and non-critical components into a data element for further log processing by the indexing, pipeline, and GUI, and wherein a critical component comprises a component that requires at least one of node affinity or log perseverance.
9 . The method of claim 8 wherein at least one of the critical and non-critical components comprise at least one of system elements or services that implement containerized applications comprising at least one of a Data Domain container running deduplication and compression processes, a cloud-native data protection manager, and a scalable object storage manager.
10 . A system for providing centralized log visualization in a cluster system operated by a user and having a plurality of nodes executing containerized applications, and comprising:
a collector collecting logs for each node of a plurality of nodes in the cluster system, wherein the logs record transactions of components of the system;
an indexer indexing the collected logs by assigning a unique index to each log file of a plurality of log files containing the logs to generate indexed log files;
a pipeline processing each indexed log file to transform unstructured log information into structured log data; and
a graphical user interface (GUI) displaying the structured log data for visual analysis by a user to debug an issue related to a corresponding log message, wherein the cluster system comprises a Santorini network processing containerized data utilizing a Kubernetes-based framework, and wherein the cluster network comprises part of a Data Domain deduplication backup system performing backup and restore operations for the nodes, and further wherein the components comprise at least one of system elements or services that implement containerized applications comprising at least one of a Data Domain container running deduplication and compression processes, a cloud-native data protection manager, and a scalable object storage manager.
11 . The system of claim 10 wherein the GUI displays a histogram of logged event occurrences over a user-defined period of time.
12 . The system of claim 11 wherein the GUI further displays detailed textual information of each logged event comprising at least date of event, log identifier, log message, and log file path.
13 . The system of claim 10 further comprising an interface component sending the structured log data to a debugger process to debug the issue, and wherein the debugger process is executed at one of: a resident site of the cluster system operated by the user, or a vendor site providing debugging services.
14 . The system of claim 13 wherein the logs are collected for events including system changes, authorization activities, privileged access events, and audit related activities.
15 . The system of claim 14 wherein the system changes comprise component failures and changes to availability, configuration, or source code; the authorization activities comprise login or access failures and access provisioning; the privileged access events comprise using a superuser status or running an administrator console; and the audit-related activities comprise changing event logging configurations, deleting event logs, or audit log failures.
16 . The system of claim 10 further comprising:
a component writing logs for critical components to dedicated persistent volumes for each respective component, and writing logs for non-critical components to a central persistent volume (PV); and
a combiner combining the collected logs for the critical and non-critical components into a data element for further log processing by the indexing, pipeline, and GUI, and wherein a critical component comprises a component that requires at least one of node affinity or log perseverance.
17 . A non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method of processing logs for centralized log visualization in a cluster system operated by a user and having a plurality of nodes executing containerized applications, by:
collecting log files containing logs for each node of a plurality of nodes in the cluster system, wherein the logs record transactions of components of the system;
indexing the collected logs by assigning a unique index to each log file to generate indexed log files;
processing each indexed log file through a pipeline to transform unstructured log information into structured log data; and
displaying the structured log data through a graphical user interface (GUI) for visual analysis by a user to debug an issue related to a corresponding log message, wherein the cluster system comprises a Santorini network processing containerized data utilizing a Kubernetes-based framework, and wherein the cluster network comprises part of a Data Domain deduplication backup system performing backup and restore operations for the nodes, and further wherein the GUI displays a histogram of logged event occurrences over a user-defined period of time, and yet further wherein the GUI further displays detailed textual information of each logged event comprising at least date of event, log identifier, log message, and log file path.