Training and using a memory failure prediction model
The disclosure herein describes training and using an uncorrectable error (UE) state prediction model based on telemetry error data. Sets of UE state labels and non-UE state labels are generated from a first set of collected telemetry data, wherein the UE state labels each reference a UE and telemetry data of an interval prior to the referenced UE. Statistical features are extracted from telemetry data of the sets of UE state labels and non-UE state labels, and the extracted statistical features are used to train a UE state prediction model. A second set of collected telemetry data is obtained, and a UE event is predicted based on the second set of collected telemetry data using the trained UE state prediction model. A preventative operation is performed on a memory page of the system based on the predicted UE event, whereby the predicted UE event is prevented from occurring.
1. A system comprising:
at least one processor; and
at least one memory comprising computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the at least one processor to:
generate a set of uncorrectable error (UE) state labels based on a first set of collected telemetry data, wherein each UE state label includes a reference to a UE event and telemetry data of a time interval prior to the referenced UE event;
generate a set of non-UE state labels based on the collected telemetry data, wherein each non-UE state label includes a reference to a correctable error (CE) event and telemetry data of a time interval, prior to the referenced CE event, that does not include a UE event;
extract statistical features from telemetry data of the set of UE state labels and from telemetry data of the set of non-UE state labels, wherein extracting the statistical features includes extracting statistical features associated with UE state labels and non-UE state labels based on a plurality of different memory hierarchy levels, wherein the plurality of memory hierarchy levels includes at least one of the following: memory page levels, memory row levels, or node levels;
train a UE state prediction model using the set of UE state labels, the set of non-UE state labels, and the extracted statistical features;
obtain a second set of collected telemetry data;
predict a UE event based on the second set of collected telemetry data using the trained UE state prediction model, wherein the predicted UE event is associated with a memory page of a system; and
perform a preventative operation on the memory page of the system, whereby the predicted UE event is prevented from occurring.
2. The system of claim 1 , wherein extracting the statistical features includes:
extracting statistical features associated with UE state labels and non-UE state labels based on a plurality of different time intervals; andextracting statistical features associated with UE state labels and non-UE state labels based on a plurality of different memory hierarchy levels, wherein the plurality of memory hierarchy levels includes at least one of the following: memory page levels, memory row levels, or node levels.
3. The system of claim 2 , wherein:
training the UE state prediction model includes training the UE state prediction model to predict UE events associated with the plurality of different memory hierarchy levels using the extracted statistical features associated with the plurality of different memory hierarchy levels; and
wherein-performing the preventative operation on the memory page of the system further includes performing a preventative operation based on a memory hierarchy level with which the predicted UE event is associated.
4. The system of claim 1 , wherein the telemetry data of the time interval prior to the referenced UE event of each UE state label excludes telemetry data of a sub-interval immediately before the referenced UE event, wherein the sub-interval is of an error prevention interval length.
5. The system of claim 4 , wherein the error prevention interval length is one hour, whereby the UE state prediction model is trained to predict UE events that are likely to occur in greater than or equal to one hour.
6. The system of claim 1 , wherein training the UE state prediction model further includes:
forming a set of training data based on the set of UE state labels, the set of non-UE state labels, and the extracted statistical features;
dividing the set of training data into a training subset and a testing subset, wherein data of the training subset and data of the testing subset include exclusive node data;
training a ML model using the training subset; and
evaluating the trained ML model using the testing subset, whereby the trained UE state prediction model is based on a version of the trained ML model that meets evaluation requirements.
7. The system of claim 6 , wherein:
training the UE state prediction model further includes:
selecting a subset of extracted features with high average information gain based on training the ML model; and
extracting the selected subset of features from the testing subset; and
wherein-evaluating the trained ML model includes evaluating the trained ML model using the selected subset of features from the testing subset.
8. A computerized method comprising:
generating, by a processor, a set of uncorrectable error (UE) state labels based on a first set of collected telemetry data, wherein each UE state label references-includes a reference to a UE event and telemetry data of a time interval prior to the referenced UE event;
generating, by the processor, a set of non-UE state labels based on the collected telemetry data, wherein each non-UE state label references-includes a reference to a correctable error (CE) event and telemetry data of a time interval, prior to the referenced CE event, that does not include a UE event;
extracting, by the processor, statistical features from telemetry data of the set of UE state labels and from telemetry data of the set of non-UE state labels, wherein extracting the statistical features includes extracting statistical features associated with UE state labels and non-UE state labels based on a plurality of different memory hierarchy levels, wherein the plurality of memory hierarchy levels includes at least one of the following: memory page levels, memory row levels, or node levels;
training, by the processor, a UE state prediction model using the set of UE state labels, the set of non-UE state labels, and the extracted statistical features;
obtaining, by the processor, a second set of collected telemetry data;
predicting, by the processor, a UE event based on the second set of collected telemetry data using the trained UE state prediction model, wherein the predicted UE event is associated with a memory page of a system; and
performing, by the processor, a preventative operation on the memory page of the system, whereby the predicted UE event is prevented from occurring.
9. The computerized method of claim 8 , wherein extracting the statistical features includes:
extracting statistical features associated with UE state labels and non-UE state labels based on a plurality of different time intervals; and extracting statistical features associated with UE state labels and non UE state labels based on a plurality of different memory hierarchy levels, wherein the plurality of memory hierarchy levels includes at least one of the following: memory page levels, memory row levels, or node levels.
10. The computerized method of claim 9 , wherein:
training the UE state prediction model includes training the UE state prediction model to predict UE events associated with the plurality of different memory hierarchy levels using the extracted statistical features associated with the plurality of different memory hierarchy levels; and
wherein-performing the preventative operation on the memory page of the system further includes performing a preventative operation based on a memory hierarchy level with which the predicted UE event is associated.
11. The computerized method of claim 8 , wherein the telemetry data of the time interval prior to the referenced UE event of each UE state label excludes telemetry data of a sub-interval immediately before the referenced UE event, wherein the sub-interval is of an error prevention interval length.
12. The computerized method of claim 11 , wherein the error prevention interval length is one hour, whereby the UE state prediction model is trained to predict UE events that are likely to occur in greater than or equal to one hour.
13. The computerized method of claim 8 , wherein training the UE state prediction model further includes:
forming a set of training data based on the set of UE state labels, the set of non-UE state labels, and the extracted statistical features;
dividing the set of training data into a training subset and a testing subset, wherein data of the training subset and data of the testing subset include exclusive node data;
training a ML model using the training subset; and
evaluating the trained ML model using the testing subset, whereby the trained UE state prediction model is based on a version of the trained ML model that meets evaluation requirements.
14. The computerized method of claim 13 , wherein:
training the UE state prediction model further includes:
selecting a subset of extracted features with high average information gain based on training the ML model; and
extracting the selected subset of features from the testing subset; and
wherein evaluating the trained ML model includes evaluating the trained ML model using the selected subset of features from the testing subset.
15. One or more computer storage media having computer-executable instructions that, upon execution by a processor, cause the processor to at least:
generate a set of uncorrectable error (UE) state labels based on a first set of collected telemetry data, wherein each UE state label references includes a reference to a UE event and telemetry data of a time interval prior to the referenced UE event;
generate a set of non-UE state labels based on the collected telemetry data, wherein each non-UE state label references-includes a reference to a correctable error (CE) event and telemetry data of a time interval, prior to the referenced CE event, that does not include a UE event;
extract statistical features from telemetry data of the set of UE state labels and from telemetry data of the set of non-UE state labels, wherein extracting the statistical features includes extracting statistical features associated with UE state labels and non-UE state labels based on a plurality of different memory hierarchy levels, wherein the plurality of memory hierarchy levels includes at least one of the following: memory page levels, memory row levels, or node levels;
train a UE state prediction model using the set of UE state labels, the set of non-UE state labels, and the extracted statistical features;
obtain a second set of collected telemetry data;
predict a UE event based on the second set of collected telemetry data using the trained UE state prediction model, wherein the predicted UE event is associated with a memory page of a system; and
perform a preventative operation on the memory page of the system, whereby the predicted UE event is prevented from occurring.
16. The one or more computer storage media of claim 15 , wherein extracting the statistical features includes:
extracting statistical features associated with UE state labels and non-UE state labels based on a plurality of different time intervals; and extracting statistical features associated with UE state labels and non UE state labels based on a plurality of different memory hierarchy levels, wherein the plurality of memory hierarchy levels includes at least one of the following: memory page levels, memory row levels, or node levels.
17. The one or more computer storage media of claim 16 , wherein:
training the UE state prediction model includes training the UE state prediction model to predict UE events associated with the plurality of different memory hierarchy levels using the extracted statistical features associated with the plurality of different memory hierarchy levels; and
wherein performing the preventative operation on the memory page of the system further includes performing a preventative operation based on a memory hierarchy level with which the predicted UE event is associated.
18. The one or more computer storage media of claim 15 , wherein the telemetry data of the time interval prior to the referenced UE event of each UE state label excludes telemetry data of a sub-interval immediately before the referenced UE event, wherein the sub-interval is of an error prevention interval length.
19. The one or more computer storage media of claim 15 , wherein training the UE state prediction model further includes:
forming a set of training data based on the set of UE state labels, the set of non-UE state labels, and the extracted statistical features;
dividing the set of training data into a training subset and a testing subset, wherein data of the training subset and data of the testing subset include exclusive node data;
training a ML model using the training subset; and
evaluating the trained ML model using the testing subset, whereby the trained UE state prediction model is based on a version of the trained ML model that meets evaluation requirements.
20. The one or more computer storage media of claim 19 , wherein:
training the UE state prediction model further includes:
selecting a subset of extracted features with high average information gain based on training the ML model; and
extracting the selected subset of features from the testing subset; and
wherein-evaluating the trained ML model includes evaluating the trained ML model using the selected subset of features from the testing subset.