Prediction method and apparatus for faulty GPU, based on prediction model electronic device and storage medium
The present disclosure provides a prediction method and an apparatus for a faulty GPU, an electronic device and a storage medium. The method includes: acquiring parameter information of each GPU in a plurality of GPUs to obtain a parameter information set; inputting the parameter information set into a plurality of pre-trained prediction models to obtain a prediction result corresponding to each prediction model; and determining a faulty GPU from the plurality of GPUs according to the prediction result.
1. A prediction method for a faulty GPU, comprising:
acquiring parameter information of each GPU in a plurality of GPUs to obtain a parameter information set;
inputting the parameter information set into a plurality of pre-trained prediction models to obtain a prediction result corresponding to each prediction model; and
determining a faulty GPU from the plurality of GPUs according to the prediction result,
wherein the plurality of prediction models comprise a first prediction model and a second prediction model,
inputting the parameter information set into the plurality of pre-trained prediction models to obtain the prediction result corresponding to each prediction model comprises:
inputting the parameter information set into the first prediction model to obtain a first prediction result corresponding to the first prediction model, wherein the first prediction result comprises a first fault rate corresponding to each GPU;
selecting one or more first target GPUs from the plurality of GPUs according to the first fault rate corresponding to each GPU; and
inputting parameter information of the one or more first target GPUs into the second prediction model to obtain a second prediction result corresponding to the second prediction model, wherein the second prediction result comprises a second fault rate corresponding to each GPU in the one or more first target GPUs; and
determining the faulty GPU from the plurality of GPUs according to the prediction results comprises:
determining the faulty GPU from the one or more first target GPUs according to the second fault rate corresponding to each GPU in the one or more first target GPUs.
2. The method according to claim 1 , wherein
selecting the one or more first target GPUs from the plurality of GPUs according to the first fault rate corresponding to each GPU comprises:
determining top m GPUs with higher first fault rates in the first prediction result as the one or more first target GPUs, wherein m is a positive integer; and
determining the faulty GPU from the one or more first target GPUs according to the second fault rate corresponding to each GPU in the one or more first target GPUs comprises:
determining top n first target GPUs with higher second fault rates in the second prediction result as the faulty GPU, wherein n is a positive integer.
3. The method according to claim 1 , wherein the first prediction model is a 1D-CNN model and the second prediction model is an MLP model;
or, the first prediction model is an MLP model and the second prediction model is a 1D-CNN model.
4. The method according to claim 1 , wherein acquiring the parameter information of each GPU in the plurality of GPUs to obtain the parameter information set comprises:
acquiring the parameter information of each GPU in the plurality of GPUs according to a preset cycle to obtain the parameter information set, wherein a cycle duration of the preset cycle is greater than or equal to a specified duration.
5. The method according to claim 1 , wherein before acquiring the parameter information of each GPU in the plurality of GPUs to obtain the parameter information set, the method further comprises:
acquiring a parameter information sample of a GPU;
performing sliding-sampling on the parameter information sample through a preset sliding window to obtain a first sample; and
performing model training based on the first sample to obtain a plurality of prediction models.
6. The method according to claim 5 , further comprising:
in a case where a service time of a currently used prediction model is longer than a preset duration, acquiring a new parameter information sample of the GPU, and performing sliding-sampling on the new parameter information sample through the preset sliding window to obtain a second sample; and
performing model training based on the second sample to obtain the plurality of prediction models.
7. A non-transitory computer-readable medium, comprising a computer program, wherein, when the computer program is executed by a processing apparatus, steps of the method according to claim 1 are implemented.
8. An electronic device, comprising:
a storage apparatus storing one or more computer programs; and
one or more processing apparatuses, configured to execute the one or more computer programs on the storage apparatus to implement steps of the method according to claim 1 .
9. The electronic device according to claim 8 , wherein acquiring the parameter information of each GPU in the plurality of GPUs to obtain the parameter information set comprises:
acquiring the parameter information of each GPU in the plurality of GPUs according to a preset cycle to obtain the parameter information set, wherein a cycle duration of the preset cycle is greater than or equal to a specified duration.
10. The electronic device according to claim 8 , wherein
selecting the one or more first target GPUs from the plurality of GPUs according to the first fault rate corresponding to each GPU comprises:
determining top m GPUs with higher first fault rates in the first prediction result as the one or more first target GPUs, wherein m is a positive integer; and
determining the faulty GPU from the one or more first target GPUs according to the second fault rate corresponding to each GPU in the one or more first target GPUs comprises:
determining top n first target GPUs with higher second fault rates in the second prediction result as the faulty GPU, wherein n is a positive integer.
11. The electronic device according to claim 8 , wherein the first prediction model is a 1D-CNN model and the second prediction model is an MLP model;
or, the first prediction model is an MLP model and the second prediction model is a 1D-CNN model.
12. A prediction method for a faulty GPU, comprising:
acquiring parameter information of each GPU in a plurality of GPUs to obtain a parameter information set;
inputting the parameter information set into a plurality of pre-trained prediction models to obtain a prediction result corresponding to each prediction model; and
determining a faulty GPU from the plurality of GPUs according to the prediction result,
wherein the prediction models comprise a first prediction model, a second prediction model and a third prediction model,
inputting the parameter information set into the plurality of pre-trained prediction models to obtain the prediction result corresponding to each prediction model comprises:
inputting the parameter information set into the first prediction model to obtain a first prediction result corresponding to the first prediction model, wherein the first prediction result comprises a first fault rate corresponding to each GPU;
inputting the parameter information set into the second prediction model to obtain a second prediction result corresponding to the second prediction model, wherein the second prediction result comprises a second fault rate corresponding to each GPU; and
inputting the parameter information set into the third prediction model to obtain a third prediction result corresponding to the third prediction model, wherein the third prediction result comprises a third fault rate corresponding to each GPU; and
determining the faulty GPU from the plurality of GPUs according to the prediction result comprises:
selecting one or more first target GPUs from the plurality of GPUs according to the first fault rate corresponding to each GPU;
selecting a second target GPU from the plurality of GPUs according to the second fault rate corresponding to each GPU;
selecting a third target GPU from the plurality of GPUs according to the third fault rate corresponding to each GPU; and
determining a same GPU among the one or more first target GPUs, the second target GPU and the third target GPU as the faulty GPU.
13. The method according to claim 12 , wherein
selecting the one or more first target GPUs from the plurality of GPUs according to the first fault rate corresponding to each GPU comprises:
determining top m GPUs with higher first fault rates in the first prediction result as the one or more first target GPUs, wherein m is a positive integer;
selecting the second target GPU from the plurality of GPUs according to the second fault rate corresponding to each GPU comprises:
determining top n GPUs with higher second fault rates in the second prediction result as the second target GPU, wherein n is a positive integer; and
selecting the third target GPU from the plurality of GPUs according to the third fault rate corresponding to each GPU comprises:
determining top k GPUs with higher third fault rates in the third prediction result as the third target GPU, wherein k is a positive integer.
14. The method according to claim 12 , wherein the first prediction model is a 1D-CNN model, the second prediction model is an MLP model, and the third prediction model is a GB model.
15. An electronic device, comprising:
a storage apparatus storing one or more computer programs; and
one or more processing apparatuses, configured to execute the one or more computer programs on the storage apparatus to implement steps of the method according to claim 12 .
16. The electronic device according to claim 15 , wherein
selecting the one or more first target GPUs from the plurality of GPUs according to the first fault rate corresponding to each GPU comprises:
determining top m GPUs with higher first fault rates in the first prediction result as the one or more first target GPUs, wherein m is a positive integer;
selecting the second target GPU from the plurality of GPUs according to the second fault rate corresponding to each GPU comprises:
determining top n GPUs with higher second fault rates in the second prediction result as the second target GPU, wherein n is a positive integer; and
selecting the third target GPU from the plurality of GPUs according to the third fault rate corresponding to each GPU comprises:
determining top k GPUs with higher third fault rates in the third prediction result as the third target GPU, wherein k is a positive integer.
17. The electronic device according to claim 15 , wherein the first prediction model is a 1D-CNN model, the second prediction model is an MLP model, and the third prediction model is a GB model.
18. A non-transitory computer-readable medium, comprising a computer program, wherein, when the computer program is executed by a processing apparatus, steps of the method according to claim 12 are implemented.