Optimizing data placement based on data temperature and lifetime prediction
A method for optimizing data storage includes obtaining a data object for storage at memory hardware in communication with data processing hardware. The memory hardware includes a plurality of storage devices, each storage device of the plurality of storage devices including storage parameters different from each other storage device of the plurality of storage devices. The method also includes determining one or more data object parameters associated with the data object and predicting, using a model and the data object parameters and the storage parameters, an object temperature representative of a frequency of access for the data object and an object lifetime representative of an amount of time the data object is to be stored. The method further includes selecting, using the predicted object temperature and object lifetime, one of the storage devices, and storing the data object at the selected one of the storage devices.
1 . A computer-implemented method when executed by data processing hardware that causes the data processing hardware to perform operations comprising:
obtaining a data object for storage at memory hardware in communication with the data processing hardware, the memory hardware comprising a plurality of storage devices, each storage device of the plurality of storage devices comprising storage parameters different from each other storage device of the plurality of storage devices;
prior to storing the data object at the memory hardware:
determining one or more data object parameters associated with the data object;
predicting, using a model, based on the one or more data object parameters and the storage parameters of the plurality of storage devices:
an object temperature of the data object, the object temperature representative of a frequency of access for the data object; and
an object lifetime of the data object, the object lifetime representative of an amount of time the data object is to be stored; and
selecting, based on the predicted object temperature of the data object, the predicted object lifetime of the data object, and the storage parameters of the plurality of storage devices, one of the storage devices of the plurality of storage devices; and
storing the data object at the selected one of the storage devices.
2 . The method of claim 1 , wherein the storage parameters comprise at least one of:
a geographical location;
network connectivity;
input/output density; or
data erasure characteristics.
3 . The method of claim 1 , wherein the data object parameters comprise at least one of:
a data owner;
an object name;
an object size;
a creation time;
an object age; or
an object creation mechanism.
4 . The method of claim 1 , wherein predicting the object temperature and the object lifetime of the data object comprises classifying the data object using Bayesian Inference.
5 . The method of claim 1 , wherein predicting the object temperature and the object lifetime of the data object comprises generating a prediction using one of a machine learning classification algorithm or a machine learning regression algorithm.
6 . The method of claim 1 , wherein selecting the one of the storage devices of the plurality of storage devices comprises performing a cost-benefit analysis.
7 . The method of claim 6 , wherein the cost-benefit analysis comprises a per-byte cost of each storage device of the plurality of storage devices.
8 . The method of claim 1 , wherein the operations further comprise, after storing the data object at the selected one of the storage devices:
predicting, using updated data object parameters and the storage parameters:
an updated object temperature of the data object; and
an updated object lifetime of the data object; and
selecting, using the updated object temperature of the data object and the updated object lifetime of the data object, a second one of the storage devices of the plurality of storage devices.
9 . The method of claim 1 , where the operations further comprise, prior to predicting the object temperature of the data object and the object lifetime of the data object, training the model using historical temperature and lifetime training samples.
10 . The method of claim 9 , wherein the historical temperature and lifetime training samples are biased using a Kaplan-Meier estimator.
11 . A system comprising:
data processing hardware; and
memory hardware comprising a plurality of storage devices, each storage device of the plurality of storage devices comprising storage parameters different from each other storage device of the plurality of storage devices, the memory hardware in communication with the data processing hardware and storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations comprising:
obtaining a data object for storage at the memory hardware;
prior to storing the data object at the memory hardware:
determining one or more data object parameters associated with the data object;
predicting, using a model, based on the one or more data object parameters and the storage parameters of the plurality of storage devices:
an object temperature of the data object, the object temperature representative of a frequency of access for the data object; and
an object lifetime of the data object, the object lifetime representative of an amount of time the data object is to be stored; and
selecting, based on the predicted object temperature of the data object, the predicted object lifetime of the data object, and the storage parameters of the plurality of storage devices, one of the storage devices of the plurality of storage devices; and
storing the data object at the selected one of the storage devices.
12 . The system of claim 11 , wherein the storage parameters comprises at least one of:
a geographical location;
network connectivity;
input/output density; or
data erasure characteristics.
13 . The system of claim 11 , wherein the data object parameters comprise at least one of:
a data owner;
an object name;
an object size;
a creation time;
an object age; or
an object creation mechanism.
14 . The system of claim 11 , wherein predicting the object temperature and the object lifetime of the data object comprises classifying the data object using Bayesian Inference.
15 . The system of claim 11 , wherein predicting the object temperature and the object lifetime of the data object comprises generating a prediction using one of a machine learning classification algorithm or a machine learning regression algorithm.
16 . The system of claim 11 , wherein selecting the one of the storage devices of the plurality of storage devices comprises performing a cost-benefit analysis.
17 . The system of claim 16 , wherein the cost-benefit analysis comprises a per-byte cost of each storage device of the plurality of storage devices.
18 . The system of claim 11 , wherein the operations further comprise, after storing the data object at the selected one of the storage devices:
predicting, using updated data object parameters and the storage parameters:
an updated object temperature of the data object; and
an updated object lifetime of the data object; and
selecting, using the updated object temperature of the data object and the updated object lifetime of the data object, a second one of the storage devices of the plurality of storage devices.
19 . The system of claim 11 , where the operations further comprise, prior to predicting the object temperature of the data object and the object lifetime of the data object, training the model using historical temperature and lifetime training samples.
20 . The system of claim 19 , wherein the historical temperature and lifetime training samples are biased using a Kaplan-Meier estimator.