Proactive operating system memory unit replacement based on namespace health
A computer-implemented method includes receiving diagnostic data about a plurality of operating system (OS) memory units of a distributed computing system. The method further includes defining a failure state for predicting whether at least one of the plurality of OS memory units will experience a failure. The method further includes predicting whether the at least one of the plurality of OS memory units will experience the failure based at least in part on the failure state and the diagnostic data about the plurality of OS memory units. The method further includes causing the at least one of the plurality of OS memory units predicted to experience the failure to be replaced with at least one replacement OS memory unit without impacting a namespace availability of the distributed computing system.
1 . A computer-implemented method comprising:
receiving diagnostic data about a plurality of operating system (OS) memory units of a distributed computing system, the diagnostic data comprises key-value pairs of metrics;
converting, for each of the key-value pairs of metrics, a value of a key-value pair into an array of numbers, quantizing the array into a vector using k-means clustering, classifying the vector, and storing, when classified within an unhealthy cluster, a distance to a centroid of the unhealthy cluster within the value of a key-value store;
defining a failure state for predicting whether at least one of the plurality of OS memory units will experience a failure;
predicting whether the at least one of the plurality of OS memory units will experience the failure based at least in part on the failure state and the diagnostic data about the plurality of OS memory units, wherein the predicting comprises determining that the at least one of the plurality of OS memory units is likely to fail responsive to the distance to the centroid of the unhealthy cluster being below a threshold distance; and
causing the at least one of the plurality of OS memory units predicted to experience the failure to be replaced with at least one replacement OS memory unit without impacting a namespace availability of the distributed computing system.
2 . The computer-implemented method of claim 1 , wherein receiving the diagnostic data about the plurality of OS memory units comprises collecting the diagnostic data about the plurality of OS memory units.
3 . The computer-implemented method of claim 2 , wherein the collecting is performed on a periodic basis.
4 . The computer-implemented method of claim 1 , the diagnostic data comprises self-monitoring, analysis, and reporting technology (SMART) data.
5 . The computer-implemented method of claim 1 , wherein causing the at least one of the plurality of OS memory units predicted to experience the failure to be replaced prevents the distributed computing system from entering a hang state.
6 . The computer-implemented method of claim 1 , wherein causing the at least one of the plurality of OS memory units predicted to experience the failure to be replaced prevents the distributed computing system from experiencing performance degradation.
7 . The computer-implemented method of claim 1 , wherein the failure state comprises a threshold, wherein the at least one of the plurality of OS memory units is predicted to experience the failure responsive to a metric about the at least one of the plurality of OS memory units exceeding the threshold.
8 . The computer-implemented method of claim 1 , further comprising, responsive to predicting that the at least one of the plurality of OS memory units will experience the failure, backing up data and logs stored on the at least one of the plurality of OS memory units predicted to experience the failure to at least one of a local storage vault and a management vault and synchronizing the data to the at least one replacement OS memory unit.
9 . A system comprising:
a memory comprising computer readable instructions; and
a processing device for executing the computer readable instructions, the computer readable instructions controlling the processing device to perform operations comprising:
receiving diagnostic data about a plurality of operating system (OS) memory units of a distributed computing system, the diagnostic data comprises key-value pairs of metrics;
converting, for each of the key-value pairs of metrics, a value of a key-value pair into an array of numbers, quantizing the array into a vector using k-means clustering, classifying the vector, and storing, when classified within an unhealthy cluster, a distance to a centroid of the unhealthy cluster within the value of a key-value store;
defining a failure state for predicting whether at least one of the plurality of OS memory units will experience a failure;
predicting whether the at least one of the plurality of OS memory units will experience the failure based at least in part on the failure state and the diagnostic data about the plurality of OS memory units, wherein the predicting comprises determining that the at least one of the plurality of OS memory units is likely to fail responsive to the distance to the centroid of the unhealthy cluster being below a threshold distance; and
causing the at least one of the plurality of OS memory units predicted to experience the failure to be replaced with at least one replacement OS memory unit without impacting a namespace availability of the distributed computing system.
10 . The system of claim 9 , wherein receiving the diagnostic data about the plurality of OS memory units comprises collecting the diagnostic data about the plurality of OS memory units.
11 . The system of claim 10 , wherein the collecting is performed on a periodic basis.
12 . The system of claim 9 , the diagnostic data comprises self-monitoring, analysis, and reporting technology (SMART) data.
13 . The system of claim 9 , wherein causing the at least one of the plurality of OS memory units predicted to experience the failure to be replaced prevents the distributed computing system from entering a hang state.
14 . The system of claim 9 , wherein causing the at least one of the plurality of OS memory units predicted to experience the failure to be replaced prevents the distributed computing system from experiencing performance degradation.
15 . The system of claim 9 , wherein the failure state comprises a threshold, wherein the at least one of the plurality of OS memory units is predicted to experience the failure responsive to a metric about the at least one of the plurality of OS memory units exceeding the threshold.
16 . The system of claim 9 , wherein the operations further comprise, responsive to predicting that the at least one of the plurality of OS memory units will experience the failure, backing up data and logs stored on the at least one of the plurality of OS memory units predicted to experience the failure to at least one of a local storage vault and a management vault and synchronizing the data to the at least one replacement OS memory unit.
17 . A computer program product comprising:
a set of one or more computer-readable storage media;
program instructions, collectively stored in the set of one or more storage media, for causing a processor set to perform the following computer operations:
receiving diagnostic data about a plurality of operating system (OS) memory units of a distributed computing system, the diagnostic data comprises key-value pairs of metrics;
converting, for each of the key-value pairs of metrics, a value of a key-value pair into an array of numbers, quantizing the array into a vector using k-means clustering, classifying the vector, and storing, when classified within an unhealthy cluster, a distance to a centroid of the unhealthy cluster within the value of a key-value store;
defining a failure state for predicting whether at least one of the plurality of OS memory units will experience a failure;
predicting whether the at least one of the plurality of OS memory units will experience the failure based at least in part on the failure state and the diagnostic data about the plurality of OS memory units, wherein the predicting comprises determining that the at least one of the plurality of OS memory units is likely to fail responsive to the distance to the centroid of the unhealthy cluster being below a threshold distance; and
causing the at least one of the plurality of OS memory units predicted to experience the failure to be replaced with at least one replacement OS memory unit without impacting a namespace availability of the distributed computing system.
18 . The computer program product of claim 17 , wherein receiving the diagnostic data about the plurality of OS memory units comprises collecting the diagnostic data about the plurality of OS memory units.
19 . The computer program product of claim 18 , wherein the collecting is performed on a periodic basis.
20 . The computer program product of claim 17 , the diagnostic data comprises self-monitoring, analysis, and reporting technology (SMART) data.