Self-healing latency issues in a cluster network using golden signal telemetry datasets
A telemetry processing system in a Kubernetes-based cluster network receives golden signal telemetry data from a plurality of pods and formats it into a structured format for validation and storage in a datastore. The golden signals include are generated during data protection operations and are used to reveal problems encountered by the pods. A latency golden signal measures a time between operation request and initiation. A threshold value is defined for the golden signal, and a latency probe is deployed in each pod to monitor the golden signal to detect when an operation exceeds the threshold value indicating a problem with an operation or data object. A self-healing process is called by the probe to initiate remedial measures to address the problem.
1 . A method of providing a self-healing Kubernetes-based cluster network having a plurality of nodes and pods by processing golden signal telemetry data, comprising:
first registering, with a hardware-based telemetry transmitter, a golden signal comprising telemetry data related to one of traffic, latency, errors, and saturation in a storage system of the network by naming a registering pod, a probe name, an endpoint, and a threshold value;
second registering, with a self-healing service, a hardware-implemented pod-based probe for the golden signal with the telemetry transmitter;
monitoring, by the self-healing service, an activity related to the golden signal in comparison with a respective threshold value; and
calling the probe when an activity value exceeds the respective threshold value as an indication of a problem condition to trigger implementation of remedial steps to address the problem condition and implement self-healing of the cluster network.
2 . The method of claim 1 further comprising:
deploying, within the probe, a debugger in the registering pod; and
performing a debugging operation in the debugger to identify the problem condition.
3 . The method of claim 2 wherein the golden signal comprises a latency golden signal for a data protection operation executed by the cluster network, and wherein the problem condition comprises a program call that is taking an excessive amount of time related to a normal condition, and further wherein the probe comprises a latency probe, and yet further wherein the latency golden signal measures a time between a moment a data protection operation is requested and when the data protection operation is initiated.
4 . The method of claim 3 wherein the cluster network comprises a Santorini network processing containerized data utilizing Kubernetes containerization mechanisms, and wherein the registering pod performs network functions and generates the golden signal as telemetry data for transmission to the self-healing service while performing data protection operations of a backup server.
5 . The method of claim 4 further comprising inputting the telemetry data to a datastore through a telemetry pipeline as input telemetry data, and wherein the telemetry pipeline implements an Open Telemetry (OTEL) protocol, and comprises a collector receiving the input telemetry data through a remote procedure call (RPC) process.
6 . The method of claim 2 further comprising:
collecting, by the debugger, details of the problem condition; and
analyzing time profile logs to identify a program call that is taking the excessive amount of time.
7 . The method of claim 6 further comprising, after the analyzing step, calling the self-healing service to perform the remedial steps to address the problem condition.
8 . The method of claim 7 wherein the problem condition comprises a continuous lock condition of a database, and wherein the remedial steps comprise:
identifying processes that lock the database;
taking a backup of the database;
restarting processes locking the database; and
re-accessing the database to ensure the problem condition is removed.
9 . The method of claim 8 wherein the data protection operation comprises one of a data backup, a data restore, or a data migration operation initiated by a deduplication backup program executed by a backup server.
10 . The method of claim 9 wherein metric datasets of the golden signal are formatted into a structured format for storage in a central datastore accessed by the telemetry transmitter, and wherein the second registering step comprises comparing a format of the golden signal received by the registering pod against a defined structured format.
11 . The method of claim 10 wherein the self-healing service is validated as subscribed to receive golden signal through a subscription process comprising:
implementing a selected transport mechanism through a Representational State Transfer Application Programming Interface (REST API); and
providing the REST API for each transport mechanism to enable subscription by the self-healing service.
12 . A method of debugging issues in a cluster network executing a backup operation processing data from a plurality of pods each periodically generating golden signal metric datasets for transmission to consumers, comprising:
registering, with a hardware-based telemetry transmitter, a latency golden signal measuring a time between a moment a data protection operation is requested and when the data protection operation is initiated, by storing a registering pod identifier, a latency probe identifier, an endpoint, and a threshold value;
detecting, by a hardware-implemented pod-based latency probe, when the threshold value is exceeded by an operation generating the golden signal indicating a problem with the operation; and
calling a self-healing process to identify and trigger implementation of remedial steps to address the problem and implement self-healing of the cluster network, and debug the problem with the operation.
13 . The method of claim 12 wherein the cluster network comprises a Santorini network processing containerized data utilizing Kubernetes containerization mechanisms, and wherein a pod performs network functions and generates the golden signal as telemetry data for transmission to the self-healing service while performing data protection operations of a backup server.
14 . The method of claim 13 wherein the data protection operations comprise at least one of a data backup, data restore, or data migration operation initiated by a deduplication backup program executed by the backup server, and wherein the problem comprises a program call that is taking an excessive amount of time related to a normal condition.
15 . The method of claim 14 further comprising, after an analyzing step, calling the self-healing service to perform remedial steps to address the problem condition.
16 . The method of claim 15 wherein the problem condition comprises a continuous lock condition of a database, and wherein the remedial steps comprise:
identifying processes that lock the database;
taking a backup of the database;
restarting processes locking the database; and
re-accessing the database to ensure the problem condition is removed.