Data management, reduction and sampling schemes for storage device failure
In a method for training a machine learning model, the method includes: segmenting, by a processor, a dataset from a database into one or more datasets based on time period windows; assigning, by the processor, one or more weighted values to the one or more datasets according to the time period windows of the one or more datasets; generating, by the processor, a training dataset from the one or more datasets according to the one or more weighted values; and training, by the processor, the machine learning model using the training dataset.
1. A method for training a machine learning model, the method comprising:
segmenting, by a processor, a dataset from a database into one or more datasets based on time period windows;
assigning, by the processor, one or more weighted values to the one or more datasets according to the time period windows of the one or more datasets;
generating, by the processor, a training dataset from the one or more datasets, wherein an amount of data generated from the one or more datasets is based on the one or more weighted values; and
training, by the processor, the machine learning model using the training dataset.
2. The method according to claim 1 , wherein the machine learning model comprises a solid-state drive (SSD) failure prediction model.
3. The method according to claim 1 , wherein a most recent dataset from the one or more datasets is assigned a first weighted value and a least recent dataset from the one or more datasets is assigned a second weighted value, wherein the first weighted value is greater than the second weighted value.
4. The method according to claim 3 , wherein the one or more weighted values decrease by a set amount from the first weighted value to the second weighted value.
5. The method according to claim 1 , the method further comprising:
identifying, by the processor, anomaly data in the dataset;
retrieving, by the processor, the anomaly data in the dataset; and
adding, by the processor, the anomaly data to the training dataset.
6. The method according to claim 5 , wherein the anomaly data comprises SSD failure data.
7. The method according to claim 5 , wherein the anomaly data is identified using a rule based method.
8. The method according to claim 5 , wherein the anomaly data is identified using a cluster based method.
9. The method according to claim 1 , the method further comprising
generating, by the processor, anomaly data; and
adding, by the processor, the generated anomaly data to the training dataset.
10. A data system comprising:
a database;
a processor coupled to the database; and
a memory coupled to the processor, wherein the memory stores instructions that, when executed by the processor, cause the processor to:
segment a dataset from the database into one or more datasets based on time period windows;
assign one or more weighted values to the one or more datasets according to the time period windows of the one or more datasets;
generate a training dataset from the one or more datasets, wherein an amount of data generated from the one or more datasets is based on the one or more weighted values; and
train a machine learning model using the training dataset.
11. The data system according to claim 10 , wherein the machine learning model comprises a solid-state drive (SSD) failure prediction model.
12. The data system according to claim 10 , wherein a most recent dataset from the one or more datasets is assigned a first weighted value and a least recent dataset from the one or more datasets is assigned a second weighted value, wherein the first weighted value is greater than the second weighted value.
13. The data system according to claim 12 , wherein the one or more weighted values decrease by a set amount from the first weighted value to the second weighted value.
14. The data system according to claim 10 , wherein the processor is further configured to:
identify anomaly data in the dataset;
retrieve the anomaly data in the dataset; and
add the anomaly data to the training dataset.
15. The data system according to claim 14 , wherein the anomaly data comprises SSD failure data.
16. The data system according to claim 14 , wherein the anomaly data is identified using a rule based method.
17. The data system according to claim 14 , wherein the anomaly data is identified using a cluster based method.
18. The data system according to claim 10 , wherein the processor is further configured to:
generate anomaly data; and
add the generated anomaly data to the training dataset.
19. A method for training a machine learning model, the method comprising:
identifying, by a processor, anomaly data in a dataset from a database;
generating, by the processor, additional anomaly data;
adding, by the processor, the generated anomaly data to the dataset;
identifying, by the processor, a training dataset from the dataset;
retrieving, by the processor, the training dataset from dataset; and
training, by the processor, the machine learning model using the training dataset.
20. The method according to claim 19 , wherein the machine learning model comprises a solid-state (SSD) failure prediction model.