Adaptive usage of storage resources using data source models and data source representations
Techniques are provided for adaptive usage of storage resources using data source models and data source representations generated using the data source models. One method comprises obtaining sampled data generated by sampling source data from a data source; fitting a data model to the sampled data to obtain a representation of the sampled data; obtaining a classification of the sampled data into one of multiple predefined retention models; and adapting a usage of one or more storage resources that store the retained data based on the representation and the classification. The adaptive storage resource usage may comprise, for example: (i) varying a data retention model based on an age of the sampled data; (ii) evicting cache data based on the representation; (iii) moving the retained data to a different storage tier; and (iv) determining an amount of time to store the retained data.
1 . A method, comprising:
obtaining sampled data that was generated by sampling source data from at least one data source;
obtaining a plurality of predefined retention models, wherein each of the predefined retention models specifies a different amount of retained data that is to be stored in the one or more storage resources, for at least some of the source data from the at least one data source;
fitting at least one data model to at least some of the obtained sampled data to obtain a representation of the obtained sampled data from the at least one data source;
determining, by a processor-based retention model classifier, in response to the obtaining the sampled data, an assignment of at least some of the obtained sampled data from the at least one data source to one of the plurality of predefined retention models, wherein the assignment, by the processor-based retention model classifier, to the one predefined retention model is based at least in part on one or more characteristics of the at least one data source that provides the sampled data;
storing at least a portion, corresponding to the amount of the retained data specified by the one predefined retention model, of the representation of the obtained sampled data from the at least one data source in the one or more storage resources based at least in part on the assignment to the one predefined retention model of the plurality of predefined retention models, wherein the determining is performed prior to the storing of the at least the portion of the representation of the obtained sampled data in the one or more storage resources; and
automatically adapting, by a processor-based storage resource adaptation module, using one or more resource adaptation commands sent to one or more of the storage resources, a usage of the one or more storage resources that store the retained data from the at least one data source based at least in part on the representation and the assignment to the one predefined retention model of the plurality of predefined retention models, wherein the automatically adapting the usage of the one or more storage resources comprises automatically receiving, by the processor-based storage resource adaptation module, subsequent to the storing, from the processor-based retention model classifier, an automatic selection of a different one of the plurality of predefined retention models than the one predefined retention model, wherein the automatic selection is based at least in part on: (i) one or more data life cycle rules and (ii) an age of the obtained sampled data from the at least one data source, wherein the different predefined retention model specifies a different amount of retained data than the one predefined retention model, and wherein the processor-based storage resource adaptation module automatically removes, in response to the automatic selection, at least a portion of the amount of the retained data specified by the one predefined retention model from at least one of the one or more storage resources;
wherein the method is performed by at least one processing device comprising a processor coupled to a memory.
2 . The method of claim 1 , wherein the at least one data model comprises one or more of a parametric model, a non-parametric model, a descriptive statistics model, a time series model, decision trees and an ensemble of decision trees.
3 . The method of claim 2 , wherein the fitting of the parametric model to at least some of the obtained sampled data comprises representing the obtained sampled data using a probability distribution function and determining one or more parameters of the probability distribution function.
4 . The method of claim 2 , wherein the fitting of the non-parametric model to at least some of the obtained sampled data comprises representing the obtained sampled data using one or more of a Gaussian Mixture model, a Gaussian Mixture model that captures temporal variance, and a Kernel Density Estimation model.
5 . The method of claim 2 , wherein the fitting of the descriptive statistics model to at least some of the obtained sampled data comprises recording one or more of predefined summary statistics of the representation of the obtained sampled data and a time stamp of the first and last recorded value in the sampled data.
6 . The method of claim 2 , wherein the fitting of the time series model to at least some of the obtained sampled data comprises recording the source data and corresponding time stamps generated by the at least one data source.
7 . The method of claim 2 , further comprising identifying a distribution drift based at least in part on at least one density function of one or more of the parametric model and the non-parametric model; and performing the fitting when a distribution drift is identified.
8 . The method of claim 1 , further comprising grouping a plurality of data sources into a group and representing the group using one data model.
9 . The method of claim 1 , wherein the automatically adapting the usage of the one or more storage resources further comprises one or more of (i) evicting data from a cache based at least in part on the representation; (ii) moving the retained data from the at least one data source to a different storage tier; and (iii) determining an amount of time to store the retained data from the at least one data source.
10 . The method of claim 1 , wherein the plurality of predefined retention models comprises one or more of a lossy retention model that stores a type of a probability density function, one or more parameters of the probability density function and one or more summary statistics; a subsample retention model that stores a type of a probability density function, one or more parameters of the probability density function, a time interval, one or more summary statistics and a subsample of the source data from the at least one data source; and a complete retention model that stores a type of a probability density function, one or more parameters of the probability density function, the source data from the at least one data source, and a time interval.
11 . The method of claim 1 , wherein the one or more characteristics of the at least one data source comprise one or more of a priority and a criticality of the at least one data source.
12 . An apparatus comprising:
at least one processing device comprising a processor coupled to a memory;
the at least one processing device being configured to implement the following steps:
obtaining sampled data that was generated by sampling source data from at least one data source;
obtaining a plurality of predefined retention models, wherein each of the predefined retention models specifies a different amount of retained data that is to be stored in one or more storage resources, for at least some of the source data from the at least one data source;
fitting at least one data model to at least some of the obtained sampled data to obtain a representation of the obtained sampled data from the at least one data source;
determining, by a processor-based retention model classifier, in response to the obtaining the sampled data, an assignment of at least some of the obtained sampled data from the at least one data source to one of the plurality of predefined retention models, wherein the assignment, by the processor-based retention model classifier, to the one predefined retention model is based at least in part on one or more characteristics of the at least one data source that provides the sampled data;
storing at least a portion, corresponding to the amount of the retained data specified by the one predefined retention model, of the representation of the obtained sampled data from the at least one data source in the one or more storage resources based at least in part on the assignment to the one predefined retention model of the plurality of predefined retention models, wherein the determining is performed prior to the storing of the at least the portion of the representation of the obtained sampled data in the one or more storage resources; and
automatically adapting, by a processor-based storage resource adaptation module, using one or more resource adaptation commands sent to one or more of the storage resources, a usage of the one or more storage resources that store the retained data from the at least one data source based at least in part on the representation and the assignment to the one predefined retention model of the plurality of predefined retention models, wherein the automatically adapting the usage of the one or more storage resources comprises automatically receiving, by the processor-based storage resource adaptation module, subsequent to the storing, from the processor-based retention model classifier, an automatic selection of a different one of the plurality of predefined retention models than the one predefined retention model, wherein the automatic selection is based at least in part on: (i) one or more data life cycle rules and (ii) an age of the obtained sampled data from the at least one data source, wherein the different predefined retention model specifies a different amount of retained data than the one predefined retention model, and wherein the processor-based storage resource adaptation module automatically removes, in response to the automatic selection, at least a portion of the amount of the retained data specified by the one predefined retention model from at least one of the one or more storage resources.
13 . The apparatus of claim 12 , wherein the at least one data model comprises one or more of a parametric model, a non-parametric model, a descriptive statistics model, a time series model, decision trees and an ensemble of decision trees.
14 . The apparatus of claim 12 , further comprising grouping a plurality of data sources into a group and representing the group using one data model.
15 . The apparatus of claim 12 , wherein the automatically adapting the usage of the one or more storage resources further comprises one or more of (i) evicting data from a cache based at least in part on the representation; (ii) moving the retained data from the at least one data source to a different storage tier; and (iii) determining an amount of time to store the retained data from the at least one data source.
16 . The apparatus of claim 12 , wherein the one or more characteristics of the at least one data source comprise one or more of a priority and a criticality of the at least one data source.
17 . A non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device to perform the following steps:
obtaining sampled data that was generated by sampling source data from at least one data source;
obtaining a plurality of predefined retention models, wherein each of the predefined retention models specifies a different amount of retained data that is to be stored in one or more storage resources, for at least some of the source data from the at least one data source;
fitting at least one data model to at least some of the obtained sampled data to obtain a representation of the obtained sampled data from the at least one data source;
determining, by a processor-based retention model classifier, in response to the obtaining the sampled data, an assignment of at least some of the obtained sampled data from the at least one data source to one of the plurality of predefined retention models, wherein the assignment, by the processor-based retention model classifier, to the one predefined retention model is based at least in part on one or more characteristics of the at least one data source that provides the sampled data;
storing at least a portion, corresponding to the amount of the retained data specified by the one predefined retention model, of the representation of the obtained sampled data from the at least one data source in the one or more storage resources based at least in part on the assignment to the one predefined retention model of the plurality of predefined retention models, wherein the determining is performed prior to the storing of the at least the portion of the representation of the obtained sampled data in the one or more storage resources; and
automatically adapting, by a processor-based storage resource adaptation module, using one or more resource adaptation commands sent to one or more of the storage resources, a usage of the one or more storage resources that store the retained data from the at least one data source based at least in part on the representation and the assignment to the one predefined retention model of the plurality of predefined retention models, wherein the automatically adapting the usage of the one or more storage resources comprises automatically receiving, by the processor-based storage resource adaptation module, subsequent to the storing, from the processor-based retention model classifier, an automatic selection of a different one of the plurality of predefined retention models than the one predefined retention model, wherein the automatic selection is based at least in part on: ( 1 ) one or more data life cycle rules and (ii) an age of the obtained sampled data from the at least one data source, wherein the different predefined retention model specifies a different amount of retained data than the one predefined retention model, and wherein the processor-based storage resource adaptation module automatically removes, in response to the automatic selection, at least a portion of the amount of the retained data specified by the one predefined retention model from at least one of the one or more storage resources.
18 . The non-transitory processor-readable storage medium of claim 17 , wherein the at least one data model comprises one or more of a parametric model, a non-parametric model, a descriptive statistics model, a time series model, decision trees and an ensemble of decision trees.
19 . The non-transitory processor-readable storage medium of claim 17 , wherein the automatically adapting the usage of the one or more storage resources further comprises one or more of (i) evicting data from a cache based at least in part on the representation; (ii) moving the retained data from the at least one data source to a different storage tier; and (iii) determining an amount of time to store the retained data from the at least one data source.
20 . The non-transitory processor-readable storage medium of claim 17 , wherein the one or more characteristics of the at least one data source comprise one or more of a priority and a criticality of the at least one data source.