Method, device and computer program product for error management based on a utilization rate of an accelerator device
Embodiments of the present disclosure relate to a method, device and computer program product for error management. The method comprises obtaining a utilization rate of an accelerator device in accelerator resources, and the accelerator resources comprise one or more accelerator devices. The method further comprises adjusting a detection parameter for detecting an error in the accelerator device based on the utilization rate of the accelerator device, and performing error detection for the accelerator device based on the detection parameter.
1. A method for error management, comprising:
obtaining a utilization rate of an accelerator device in accelerator resources, the accelerator resources comprising one or more accelerator devices;
adjusting a detection parameter for detecting an error in the accelerator device based on the utilization rate of the accelerator device;
performing an error detection for the accelerator device based on the detection parameter,
in accordance with determining the error of the accelerator device, performing an error recovery based on topology information of the accelerator resources, the topology information at least comprising hierarchical relations among devices, instances and sessions in the accelerator device.
2. The method of claim 1 , wherein adjusting the detection parameter for detecting the error in the accelerator device comprises:
obtaining, based a pre-established mapping table, a timeout value and a detection frequency corresponding to the utilization rate, the pre-established mapping table being obtained by experimentation or machine learning.
3. The method of claim 2 , wherein performing the error detection for the accelerator device comprises:
initiating a detection request for the accelerator device periodically according to the detection frequency;
determining a response time of the accelerator device to the detection request; and
in accordance with a determination that the response time of the accelerator device is greater than the timeout value, detecting an error of the accelerator device.
4. The method of claim 1 , wherein performing the error recovery based on topology information of the accelerator resources comprises:
determining a level of resetting for the error recovery based on an analysis for an error of the accelerator device, the level of resetting at least comprising a session level, an instance level and a device level; and
performing the resetting based on the level of resetting.
5. The method of claim 4 , wherein performing the error recovery based on topology information of the accelerator resources further comprises:
determining whether the accelerator device is recoverable based on historical error information recording historical errors of the accelerator device; and
in accordance with a determination that the accelerator device is recoverable, performing the error recovery.
6. The method of claim 1 , wherein the accelerator resources are one or more Quick Assist Technology (QAT) cards.
7. An electronic device, comprising:
a processing unit; and
a memory coupled to the processing unit and stored with instructions, the instructions, when executed by the processing unit, performing a method, the method comprising:
obtaining a utilization rate of an accelerator device in accelerator resources, the accelerator resources comprising one or more accelerator devices;
adjusting a detection parameter for detecting an error in the accelerator device based on the utilization rate of the accelerator device;
performing an error detection for the accelerator device based on the detection parameter; and
in accordance with determining the error of the accelerator device, performing an error recovery based on topology information of the accelerator resources, the topology information at least comprising hierarchical relations among devices, instances and sessions in the accelerator device.
8. The electronic device of claim 7 , wherein adjusting the detection parameter for detecting the error in the accelerator device comprises:
obtaining, based a pre-established mapping table, a timeout value and a detection frequency corresponding to the utilization rate, the pre-established mapping table being obtained by experimentation or machine learning.
9. The electronic device of claim 8 , wherein performing the error detection for the accelerator device comprises:
initiating a detection request for the accelerator device periodically according to the detection frequency;
determining a response time of the accelerator device to the detection request; and
in accordance with a determination that the response time of the accelerator device is greater than the timeout value, detecting an error of the accelerator device.
10. The electronic device of claim 7 , wherein performing the error recovery based on topology information of the accelerator resources comprises:
determining a level of resetting for the error recovery based on an analysis for an error of the accelerator device, the level of resetting at least comprising a session level, an instance level and a device level; and
performing the resetting based on the level of resetting.
11. The electronic device of claim 10 , wherein performing the error recovery based on topology information of the accelerator resources further comprises:
determining whether the accelerator device is recoverable based on historical error information recording historical errors of the accelerator device; and
in accordance with a determination that the accelerator device is recoverable, performing the error recovery.
12. A computer program product tangibly stored on a non-transitory computer readable medium and comprising computer-executable instructions, the computer-executable instructions, when executed, cause a computer to perform a method, the method comprising:
obtaining a utilization rate of an accelerator device in accelerator resources, the accelerator resources comprising one or more accelerator devices;
adjusting a detection parameter for detecting an error in the accelerator device based on the utilization rate of the accelerator device;
performing an error detection for the accelerator device based on the detection parameter; and
in accordance with determining an error in a first instance of the accelerator device, selecting a second instance of the accelerator device for failover based on a multi-dimensional information database storing topology information, historical error information and utilization rate information.
13. The computer program of claim 12 , wherein selecting the second instance of the accelerator device for failover based on the multi-dimensional information database comprises:
determining that the second instance of the accelerator device is stable based on the historical error information; and
migrating a workload in the first instance to the second instance of the accelerator device.
14. The computer program of claim 13 , wherein determining the second instance of the accelerator device comprises:
determining a plurality of stable instances of the accelerator device based on the historical error information, the plurality of stable instances comprising the second instance of the accelerator device; and
selecting the second instance of the accelerator device from the plurality of stable instances of the accelerator device based on the utilization rate information.