Encryption key rotation using dataset size and predicted threshold time based on data ingest rates
Rotating encryption keys in a size-based process using data ingestion rates to predict when a threshold size to trigger key rotation will be reached such that an equal amount of data is encrypted by different encryption keys. A machine learning (ML) model is trained based on historical data of data ingestion rates in the data storage system to predict the time that the threshold will be reached. The encryption key is then rotated at that time. A time-based rotation is triggered after a defined period to automatically rotate the key if the size-based rotation is not triggered within this period.
1 . A computer-implemented method of rotating encryption keys in a data storage system, comprising:
defining a threshold value setting an amount of data to be encrypted by a single encryption key, wherein the threshold value relates to a maximum amount of data to be encrypted by the current encryption key for storage in the data storage system;
measuring a data ingestion rate corresponding to a rate of data transmitted to or from a storage device through read/write input/output operations per second (IOPS), and wherein the current data ingestion rate comprises a percentage amount of time the storage device is busy during a defined time period and comprising a disk busy percentage parameter;
predicting a time that the threshold value will be met based on the data ingestion rate; and
performing a key rotation operation at the predicted time to use a new key to encrypt incoming data.
2 . The method of claim 1 wherein the predicting step uses a machine learning (ML) model trained using historical data ingestion rates of the data storage system.
3 . The method of claim 2 wherein the threshold value is expressed in terms of dataset size encrypted by a respective encryption key for storage in the data storage system.
4 . The method of claim 2 further comprising:
dynamically training the ML model based on updated system usage; and
checking an accuracy of the predicted time using validation data based on an actual time that the threshold was met.
5 . The method of claim 4 further comprising, if the accuracy is less than a defined accuracy threshold, re-predicting the time using the dynamically trained ML model.
6 . The method of claim 1 wherein the disk busy percentage parameter is obtained by the data storage system through a performance log file containing data use statistics compiled periodically during usage of the storage device and representing time series data for use by the ML model.
7 . The method of claim 6 wherein the storage device comprises one of a solid state drive (SSD) device and a hard disk drive (HDD), the method further comprising:
mapping the storage device to a disk name;
generating, using the performance log file, a plurality of tuples for the predicting step; and
providing the tuples to train a machine learning algorithm to detect patterns in the tuples by processing the time series data.
8 . The method of claim 7 wherein each tuple of the plurality of tuples comprises a three-part data element including the disk name, a timestamp, and a data ingestion rate at the time of the timestamp and representing a corresponding disk busy percentage value.
9 . The method of claim 7 wherein the machine learning algorithm comprises a long short-term memory algorithm that comprises a recurrent neural network process that uses a cell, an input gate, an output gate, and a forget gate.
10 . The method of claim 1 further comprising:
determining if the key rotation operation is successful in rotating to the new key from an old key; and
rotating the key automatically after a defined period of time.
11 . The method of claim 1 wherein the encryption key is provided to a key manager component by one of an external key source utilizing a Key Management Interoperability Protocol (KMIP) or an embedded key source within the data storage system, and wherein the data storage system comprises part of a deduplication backup process executed by a data storage server running a Data Domain file system (DDFS).
12 . A computer-implemented method of rotating encryption keys in a data storage system, comprising:
training a machine learning (ML) model based on historical data of data ingestion rates in the data storage system, wherein the data ingestion rates each measure a rate of data transmitted to or from a storage device through read/write input/output operations per second (IOPS), and wherein the current data ingestion rate comprises a percentage amount of time the storage device is busy during a defined time period and comprising a disk busy percentage parameter;
predicting a time to reach a threshold amount of encrypted data using the trained model, the threshold amount relates to a maximum amount of data to be encrypted by the current encryption key for storage in the data storage system; and
determining whether or the predicted time has passed, and if so, performing a key rotation operation to use a new key to encrypt incoming future data.
13 . The method of claim 12 wherein the disk busy percentage parameter is obtained by the data storage system through a performance log file containing data use statistics compiled periodically during usage of the storage device and representing time series data for use by the ML model.
14 . The method of claim 13 wherein the storage device comprises one of a solid state drive (SSD) device and a hard disk drive (HDD), the method further comprising:
mapping the storage device to a disk name;
generating, using the performance log file, a plurality of tuples for the predicting step; and
providing the tuples to train a machine learning algorithm to detect patterns in the tuples by processing the time series data, wherein each tuple of the plurality of tuples comprises a three-part data element including the disk name, a timestamp, and a data ingestion rate at the time of the timestamp and representing a corresponding disk busy percentage value.
15 . The method of claim 14 wherein the machine learning algorithm comprises a long short-term memory algorithm that comprises a recurrent neural network process that uses a cell, an input gate, an output gate, and a forget gate, and further comprising:
dynamically training the ML model based on updated system usage;
checking an accuracy of the predicted threshold value using a validation data set to detect changes in system usage; and
re-predicting, if the accuracy is less than a defined accuracy threshold, the threshold using the dynamically trained ML model.
16 . A method of rotating encryption keys encrypting respective chunks of data in a data storage system comprising:
implementing a size-based key rotation process rotating keys in accordance with maximum encrypted dataset set size as a threshold value;
automatically triggering the key rotation process based on a predicted time to reach the threshold value, wherein the predicted time is predicted using a trained machine learning (ML) algorithm using historical and present data ingestion rates; and
further triggering the key rotation after a defined time period if the automatic triggering does not occur within the defined time period; wherein a data ingestion rate measures a rate of data transmitted to or from a storage device through read/write input/output operations per second (IOPS), and wherein the current data ingestion rate comprises a percentage amount of time the storage device is busy during a defined time period and comprising a disk busy percentage parameter, and further wherein the disk busy percentage parameter is obtained by the data storage system through a performance log file containing data use statistics compiled periodically during usage of the storage device and representing time series data for use by the ML algorithm.