Debugging tools for cluster file system serviceability
A single multi-container service pod that provides a centralized cluster-aware debugger and a centralized set of debugging tools where the service pod contains one or more containers for applications executed by the cluster system. Each debugging tool of the set is associated with a respective application, such as a dump utility, a connection checker, and a log search utility. A vendor providing support for the cluster network, executes a corresponding debugging tool of the set of debugging tools to address a problem encountered by a node executing an application. Debugging the problem from the service pod impacts a debugged application on the other pods, and the service pod can trigger the debugging tools on other applications on the nodes.
1 . A method of debugging problems in a cluster system operated by a user and having a plurality of nodes executing containerized applications, comprising:
providing a centralized set of debugging tools in a service pod of a node of the cluster system, wherein the service pod contains one or more containers for applications executed by the cluster system, and wherein each debugging tool of the set is associated with a respective application, wherein the cluster system comprises a Santorini network processing containerized data utilizing a Kubernetes-based framework, and wherein the cluster network comprises part of a Data Domain deduplication backup system performing backup and restore operations for the plurality of nodes; and
executing, by a vendor providing support for the cluster network, a corresponding debugging tool of the set of debugging tools to address a problem encountered by a node executing an application, wherein each node of the cluster executes the applications from respective pods in a corresponding cluster and has installed or accesses the debugging tools, and wherein debugging the problem from the service pod impacts a debugged application on the service node and the respective pods, and yet further wherein the debugging tools comprise at least one of a dump utility, a connection checker, and a log search utility, the method further comprising providing a graphical user interface allowing service personnel to view log information, status information, or statistics from the other applications.
2 . The method of claim 1 further comprising triggering the debugging tools on other applications on the service node and the respective pods.
3 . The method of claim 2 further comprising:
executing first tools from the service pod to pause, stop, or restart the other services; and
executing second tools from the service pod to adjust configuration parameters for the other services.
4 . The method of claim 1 wherein the containerized applications comprise at least one of a Data Domain container running deduplication and compression processes, a cloud-native data protection manager, and a scalable object storage manager.
5 . The method of claim 4 wherein the statistics comprise performance and activity data for applications executed by the networked nodes, including read/write latencies, read/write throughputs, replication throughput, and garbage collection performance, and further wherein the system information comprises total storage capacity, currently utilized storage capacity, and remaining storage capacity.
6 . The method of claim 5 wherein the log information is related to at least one of: component availability state changes, component failures and errors, configuration changes, changes to source code in production, or configuration changes in a production system.
7 . A method of debugging problems in a cluster system provided by a vendor and operated by a user, and having a plurality of nodes executing containerized applications, comprising:
collecting data for each node to form collected data that is combined together in a single support bundle for transmission to the vendor, wherein the clustered network comprises a Santorini filesystem network processing containerized data utilizing a Kubernetes-based framework, and wherein the cluster network comprises part of a deduplication backup system performing backup and restore operations for the nodes, and further wherein the applications comprise at least one of a Data Domain service running deduplication and compression processes, a cloud-native data protection manager, and a scalable object storage manager;
monitoring system operation to detect occurrence of an error condition for an application, and that triggers a request from the vendor to obtain the support bundle from the user; and
providing a centralized set of debugging tools in a service pod of the cluster system, wherein each debugging tool of the set is associated with a respective containerized application for use by the vendor to debug program code causing the error condition, wherein the collected data comprises system statistics, system information, and logs for each node, and further wherein the system statistics comprise performance and activity data for applications executed by the networked nodes, including read/write latencies, read/write throughputs, replication throughput, and garbage collection performance, and yet further wherein the system information comprises total storage capacity, currently utilized storage capacity, and remaining storage capacity, and further wherein the logs comprise information related to at least one of: component availability state changes, component failures and errors, configuration changes, changes to source code in production, or configuration changes in a production system.
8 . The method of claim 7 wherein the debugging tools comprise at least one of a dump utility, a connection checker, and a log search utility, the method further comprising providing a graphical user interface allowing service personnel to view log information, status information, or statistics from the other applications.
9 . The method of claim 7 wherein each node of the cluster executes the applications from respective pods in a corresponding cluster, and wherein debugging the problem from the service pod impacts a debugged application on the service node and the respective pods.
10 . The method of claim 9 wherein each node of the cluster has installed or accesses the debugging tools.
11 . The method of claim 10 further comprising triggering the debugging tools on other applications on the service node and the respective pods.
12 . The method of claim 11 further comprising:
executing first tools from the service pod to pause, stop, or restart the other services; and
executing second tools from the service pod to adjust configuration parameters for the other services.
13 . A system for debugging problems in a cluster system operated by a user and having a plurality of nodes executing containerized applications, comprising:
a Santorini filesystem network processing containerized data utilizing a Kubernetes-based framework, and wherein the cluster network comprises part of a deduplication backup system performing backup and restore operations for the nodes, and further wherein the applications comprise at least one of a Data Domain service running deduplication and compression processes, a cloud-native data protection manager, and a scalable object storage manager;
a service pod in a node of the cluster system contains a centralized set of debugging tools, wherein the service pod further contains one or more containers for applications executed by the cluster system, and wherein each debugging tool of the set is associated with a respective application;
a collector collecting data comprising system statistics, system information, and logs for each node, and further wherein the system statistics comprise performance and activity data for applications executed by the networked nodes, including read/write latencies, read/write throughputs, replication throughput, and garbage collection performance, and yet further wherein the system information comprises total storage capacity, currently utilized storage capacity, and remaining storage capacity, and further wherein the logs comprise information related to at least one of: component availability state changes, component failures and errors, configuration changes, changes to source code in production, or configuration changes in a production system; and
a vendor system operated by a vendor providing support for the cluster network, executing a corresponding debugging tool of the set of debugging tools to address a problem encountered by a node executing an application.
14 . The system of claim 13 wherein each node of the cluster executes the applications from respective pods in a corresponding cluster, and wherein debugging the problem from the service pod impacts a debugged application on the service node and the respective pods, and further wherein each node of the cluster has installed or accesses the debugging tools, and the service pod further triggers the debugging tools on other applications on the service node and the respective pods.