Preemptive deep diagnostics and health checking of resources in disaggregated data centers

1. A method for preemptive deep diagnostics of resources in a disaggregated computing system, by a processor, comprising:

responsive to detecting a threshold breach of a recurrent event associated with a first resource of a first resource type executing a workload, generating an alert;

responsive to receiving the alert, ceasing the execution of the workload on the first resource;

identifying and invoking health check diagnostics on the first resource based on the alert and a server telemetry;

mapping results of the health check diagnostics to a set of learned failure patterns; and

predicting a potential failure of the first resource based on the mapping.

2. The method of claim 1 , further including receiving a set of preliminary monitored events from a monitoring framework, the monitoring framework monitoring the first resource; and

aggregating and correlating the set of preliminary monitored events over a period of time to identify whether one of the set of preliminary monitored events reoccurred above the threshold; wherein the recurrent event comprises the one of the set of preliminary monitored events determined to occur above the threshold.

3. The method of claim 1 , wherein the threshold is predefined for the first resource type using input by at least one of a system administrator and the learned failure patterns.

4. The method of claim 1 , responsive to determining the alert is of a certain alert type, performing a selected one of:

removing the first resource from a server which is executing the workload and assigning the first resource to a dedicated diagnostics server to perform the health check diagnostics;

assigning a second resource from an available pool of the first resource type to a server which is executing the workload while removing the first resource from the server; or

assigning a second resource from an available pool of the first resource type to a server which is executing the workload; wherein the second resource is responsible for handling workload requests from a user.

5. The method of claim 1 , further including estimating an expected time to failure (ETTF) for each resource of the first resource type based on the learned failure patterns.

6. The method of claim 1 , further including, responsive to determining that a predicted failure of the first resource is permanent, permanently failing and isolating the first resource from a pool of the first resource type.

7. The method of claim 6 , further including, responsive to determining that the predicted failure is temporary, adding the first resource back to the pool of the first resource type to fulfill future provisioning requests.

8. A system for preemptive deep diagnostics of resources in a disaggregated computing system, comprising:

a processor executing instructions stored in a memory device; wherein the processor:

responsive to detecting a threshold breach of a recurrent event associated with a first resource of a first resource type executing a workload, generates an alert;

responsive to receiving the alert, ceases the execution of the workload on the first resource;

identifies and invokes health check diagnostics on the first resource based on the alert and a server telemetry;

maps results of the health check diagnostics to a set of learned failure patterns; and

predicts a potential failure of the first resource based on the mapping.

9. The system of claim 8 , wherein the processor receives a set of preliminary monitored events from a monitoring framework, the monitoring framework monitoring the first resource; and

aggregates and correlates the set of preliminary monitored events over a period of time to identify whether one of the set of preliminary monitored events reoccurred above the threshold; wherein the recurrent event comprises the one of the set of preliminary monitored events determined to occur above the threshold.

10. The system of claim 8 , wherein the threshold is predefined for the first resource type using input by at least one of a system administrator and the learned failure patterns.

11. The system of claim 8 , wherein the processor, responsive to determining the alert is of a certain alert type, performs a selected one of:

removing the first resource from a server which is executing the workload and assigning the first resource to a dedicated diagnostics server to perform the health check diagnostics;

assigning a second resource from an available pool of the first resource type to a server which is executing the workload while removing the first resource from the server; or

assigning a second resource from an available pool of the first resource type to a server which is executing the workload; wherein the second resource is responsible for handling workload requests from a user.

12. The system of claim 8 , wherein the processor estimates an expected time to failure (ETTF) for each resource of the first resource type based on the learned failure patterns.

13. The system of claim 8 , wherein the processor, responsive to determining that a predicted failure of the first resource is permanent, permanently fails and isolates the first resource from a pool of the first resource type.

14. The system of claim 13 , wherein the processor, responsive to determining that the predicted failure is temporary, adds the first resource back to the pool of the first resource type to fulfill future provisioning requests.

15. A computer program product for preemptive deep diagnostics of resources in a disaggregated computing system, by a processor, the computer program product embodied on a non-transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:

an executable portion that, responsive to detecting a threshold breach of a recurrent event associated with a first resource of a first resource type executing a workload, generates an alert;

an executable portion that, responsive to receiving the alert, ceases the execution of the workload on the first resource;

an executable portion that identifies and invokes health check diagnostics on the first resource based on the alert and a server telemetry;

an executable portion that maps results of the health check diagnostics to a set of learned failure patterns; and

an executable portion that predicts a potential failure of the first resource based on the mapping.

16. The computer program product of claim 15 , further including an executable portion that receives a set of preliminary monitored events from a monitoring framework, the monitoring framework monitoring the first resource; and

an executable portion that aggregates and correlates the set of preliminary monitored events over a period of time to identify whether one of the set of preliminary monitored events reoccurred above the threshold; wherein the recurrent event comprises the one of the set of preliminary monitored events determined to occur above the threshold.

17. The computer program product of claim 15 , wherein the threshold is predefined for the first resource type using input by at least one of a system administrator and the learned failure patterns.

18. The computer program product of claim 15 , further including an executable portion that, responsive to determining the alert is of a certain alert type, performs a selected one of:

removing the first resource from a server which is executing the workload and assigning the first resource to a dedicated diagnostics server to perform the health check diagnostics;

assigning a second resource from an available pool of the first resource type to a server which is executing the workload while removing the first resource from the server; or

assigning a second resource from an available pool of the first resource type to a server which is executing the workload; wherein the second resource is responsible for handling workload requests from a user.

19. The computer program product of claim 15 , further including an executable portion that estimates an expected time to failure (ETTF) for each resource of the first resource type based on the learned failure patterns.

20. The computer program product of claim 15 , further including an executable portion that, responsive to determining that a predicted failure of the first resource is permanent, permanently fails and isolates the first resource from a pool of the first resource type.

21. The computer program product of claim 20 , further including an executable portion that, responsive to determining that the predicted failure is temporary, adds the first resource back to the pool of the first resource type to fulfill future provisioning requests.