Real-time predictive maintenance of hardware components using a stacked deep learning architecture on time-variant parameters combined with a dense neural network supplied with exogeneous static outputs
A system, method, and computer-readable medium are provided for a hardware component failure prediction system that can incorporate a time-series dimension as an input while also addressing issues related to a class imbalance problem associated with failure data. Embodiments utilize a double-stacked long short-term memory (DS-LSTM) deep neural network with a first layer of the DS-LSTM passing hidden cell states learned from a sequence of multi-dimensional parameter time steps to a second layer of the DS-LSTM that is configured to capture a next sequential prediction output. Output from the second layer is combined with a set of categorical variables to an input layer of a fully-connected dense neural network layer. Information generated by the dense neural network provides prediction of whether a hardware component will fail in a given future time interval.
1. A computer-implementable method for predicting failure of a hardware device, the method comprising:
acquiring telemetry data from a plurality of hardware devices of a same type;
generating a training dataset and a validation dataset from the telemetry data, wherein each dataset comprises a set of data observations over time for each device in the dataset;
training a model for predicting failure of a hardware device of the plurality of hardware devices, wherein
said training uses a modified version of the training dataset, and
said training comprises
providing cells of a first layer of a double stacked long short-term memory (DS-LSTM) network with a normalized continuous feature set of time observations for each device in the training dataset wherein each cell of the first layer of the DS-LSTM outputs a hidden cell state,
providing the hidden cell state of each first layer cell to a corresponding cell in a second layer of the DS-LSTM wherein the second layer of the DS-LSTM outputs an output vector for each device in the training dataset,
concatenating the output vector for a device with categorical parameters corresponding to the device from the training dataset to generate a concatenated result matrix for the device, and
providing the concatenated result matrix to a dense neural network (DNN) to identify continuous and categorical parameters related to device failure as an output from the DNN; and
tuning the model, wherein said tuning uses a modified version of the validation dataset to further rank continuous and categorical parameters related to device failure.
2. The method of claim 1 further comprising generating the modified versions of the training dataset and the validation dataset, wherein said generating the modified versions comprises:
ranking the set of data observations in each of the training and validation datasets in reverse chronological order;
splitting each of the training and validation datasets into a set of records associated with failed devices and passing devices; and
generating an oversampled set of observations from the set of records associated with failed devices in the training dataset to address imbalance between a number of records in the set of records associated with failed devices and passing devices.
3. The method of claim 2 , wherein said generating the oversampled set of observations from the set of records associated with failed devices in the training dataset further comprises synthetically creating repetitive samples using a moving time window.
4. The method of claim 3 , said synthetically creating repetitive samples using a moving time window further comprises:
generating an oversampled set of observations d from a actual observations such that for observation n in the set of observations, the observation is in a date range characterized by d+2−n, d+a+1−n, where d, a and n are positive integers greater than 1.
5. The method of claim 2 , wherein said generating the modified versions further comprises:
generating a set of observations from the set of records associated with failed devices in the validation dataset by further performing steps of
dividing the records associated with the failed devices in the validation dataset into d buckets, and
selecting, for each bucket, a days of observations such that for bucket n the ranks for each observation is in a range of d+2−n, d+a+1−n, where d, a and n are positive integers greater than 1.
6. The method of claim 1 further comprising:
concatenating the output from the DNN to a next concatenated result matrix to form a concatenated result matrix for a next DNN calculation.
7. The method of claim 1 , wherein each of the plurality of hardware devices is an internet of things device configured to provide the telemetry data over a network.
8. The method of claim 1 further comprising:
generating a hold-out dataset from the telemetry data, wherein
the hold-out dataset is used to determine accuracy of the model subsequent to said training and tuning, and
the hold-out dataset comprises entries distinct from the entries in the training and validation datasets.
9. A system comprising:
a processor;
a data bus coupled to the processor;
a double stacked long short-term memory (DS-LSTM) module, coupled to the processor, and having a plurality of first layer cells and corresponding plurality of second layer cells, wherein each cell of the first layer cells is configured to provide a hidden cell state to a corresponding cell of the second layer cells;
a dense neural network (DNN) module, coupled to the DS-LSTM module and the processor, and configured to receive an output from the DS-LSTM as an input; and
a non-transitory, computer-readable storage medium embodying computer program code, the non-transitory, computer-readable storage medium being coupled to the data bus, the computer program code interacting with a plurality of computer operations and comprising instructions executable by the processor and configured for:
acquiring telemetry data from a plurality of hardware devices of a same type,
generating a training dataset and a validation dataset from the telemetry data, wherein each dataset comprises a set of data observations over time for each device in the dataset,
training a model for predicting failure of a hardware device of the plurality of hardware devices, wherein
said training uses a modified version of the training dataset, and
said training comprises
providing the plurality of first layer cells of the DS-LSTM with a normalized continuous feature set of time observations for each device in the training dataset,
outputting an output vector by the second layer of the DS-LSTM for each device in the training dataset,
concatenating the output vector for a device with categorical parameters corresponding to the device from the training dataset to generate a concatenated result matrix for the device, and
providing the concatenated result matrix to the DNN module for identifying continuous and categorical parameters related to device failure, and
tuning the model, wherein said tuning uses a modified version of the validation dataset to further rank continuous and categorical parameters related to device failure.
10. The system of claim 9 , wherein the instructions executable by the processor are further configured for generating the modified versions of the training dataset and the validation dataset, wherein the instructions are configured for:
ranking the set of data observations in each of the training and validation datasets in reverse chronological order;
splitting each of the training and validation datasets into a set of records associated with failed devices and passing devices; and
generating an oversampled set of observations from the set of records associated with failed devices in the training dataset to address imbalance between a number of records in the set of records associated with failed devices and passing devices.
11. The system of claim 10 wherein the instructions for generating the oversampled set of observations from the set of records associated with failed devices in the training dataset further comprise instructions configured for synthetically creating repetitive samples using a moving time window.
12. The system of claim 11 wherein the instructions for synthetically creating repetitive samples using a moving time window further comprise instructions configured for generating an oversampled set of observations d from a actual observations such that for observation n in the set of observations, the observation is in a date range characterized by d+2−n, d+a−1−n, where d, a and n are positive integers greater than 1.
13. The system of claim 10 , wherein the instructions configured for generating the modified version of the validation dataset are further configured for
generating a set of observations from the set of records associated with failed devices in the validation dataset by further comprising instructions to
divide the records associated with the failed device in the validation dataset into d buckets, and
select, for each bucket, a days of observations such that for bucket n the ranks for each observation is in a range of d+2−n, d+a+1−n, where d, a and n are positive integers greater than 1.
14. The system of claim 9 , wherein the instructions are further configured for
generating a hold-out dataset from the telemetry data, wherein
the hold-out dataset is used to determine accuracy of the model subsequent to said training and tuning, and
the hold-out dataset comprises entries distinct from the entries in the training and validation datasets.
15. The system of claim 9 further comprising:
a network interface, coupled to the data bus and a network, wherein each of the plurality of hardware devices is an internet of things device configured to provide the telemetry data over the network.
16. A non-transitory, computer-readable storage medium embodying computer program code, the computer program code comprising computer executable instructions configured for:
acquiring telemetry data from a plurality of hardware devices of a same type;
generating a training dataset and a validation dataset from the telemetry data, wherein each dataset comprises a set of data observations over time for each device in the dataset;
training a model for predicting failure of a hardware device of the plurality of hardware devices, wherein
said training uses a modified version of the training dataset, and
said training comprises
providing cells of a first layer of a double stacked long short-term memory (DS-LSTM) network with a normalized continuous feature set of time observations for each device in the training dataset wherein each cell of the first layer of the DS-LSTM outputs a hidden cell state,
providing the hidden cell state of each first layer cell to a corresponding cell in a second layer of the DS-LSTM wherein the second layer of the DS-LSTM outputs an output vector for each device in the training dataset,
concatenating the output vector for a device with categorical parameters corresponding to the device from the training dataset to generate a concatenated result matrix for the device, and
providing the concatenated result matrix to a dense neural network (DNN) to identify continuous and categorical parameters related to device failure as an output from the DNN; and
tuning the model, wherein said tuning uses a modified version of the validation dataset to further rank continuous and categorical parameters related to device failure.
17. The non-transitory, computer-readable storage medium of claim 16 , wherein the computer executable instructions are further configured for generating the modified versions of the training dataset and the validation dataset, wherein the instructions for generating the modified versions are configured for:
ranking the set of data observations in each of the training and validation datasets in reverse chronological order;
splitting each of the training and validation datasets into a set of records associated with failed devices and passing devices; and
generating an oversampled set of observations from the set of records associated with failed devices in the training dataset to address imbalance between a number of records in the set of records associated with failed devices and passing devices.
18. The non-transitory, computer-readable storage medium of claim 17 , wherein the instructions for generating the oversampled set of observations from the set of records associated with failed devices in the training dataset further comprise instructions for synthetically creating repetitive samples using a moving time window.
19. The non-transitory, computer-readable storage medium of claim 16 , wherein the instructions for synthetically creating repetitive samples using the moving time window further comprise instructions for generating an oversampled set of observations d from a actual observations such that for observation n in the set of observations, the observation is in a date range characterized by d+2−n, d+a+1−n, where d, a and n are positive integers greater than 1.
20. The non-transitory, computer-readable storage medium of claim 16 , wherein the instructions for generating the modified versions further comprise instructions for:
generating a set of observations from the set of records associated with failed devices in the validation dataset by further performing steps of
dividing the records associated with the failed devices in the validation dataset into d buckets, and
selecting, for each bucket, a days of observations such that for bucket n the ranks for each observation is in a range of d+2−n, d+a+1−n, where d, a and n are positive integers greater than 1.