Stochastic risk scoring with counterfactual analysis for storage capacity
One example method includes accessing a dataset, selecting a list of attributes of the dataset, each of the attributes being selected based on a determination that the attribute is affecting growth of the dataset and affecting an amount of data storage space consumed by the dataset, assigning a SHAP score to each attribute, using the SHAP scores to assign respective weights to each attribute, deriving drift and shock information for the dataset, and based on the drift and shock information, calculating a risk score that a storage capacity of an asset where the dataset is stored will be exhausted within a particular time interval.
1. A method, comprising:
accessing a dataset;
selecting a list of attributes of the dataset, each of the attributes being selected based on a determination that the attribute is affecting growth of the dataset and affecting an amount of data storage space consumed by the dataset;
assigning a SHAP score to each attribute;
using the SHAP scores to assign respective weights to each attribute;
deriving drift and shock information for the dataset, and the drift and shock information is derived from the SHAP scores;
based on the drift and shock information, calculating a risk score that a storage capacity of an asset where the dataset is stored will be exhausted within a particular time interval; and
using the risk score as a basis to identify, and implement, an action to reduce a risk that the storage capacity of the asset will be exhausted within the particular time interval.
2. The method as recited in claim 1 , wherein the risk score comprises a probability that the storage capacity of the asset where the dataset is stored will be exhausted within the particular time interval.
3. The method as recited in claim 1 , wherein the drift is determined based on a weighted average of the attribute weights, and the shock is determined based on compression of data in the dataset.
4. The method as recited in claim 1 , wherein a size of the dataset changes nonlinearly over time.
5. The method as recited in claim 1 , wherein the risk score is calculated using Geometric Brownian Motion.
6. The method as recited in claim 1 , further comprising identifying one or more causal relationships among the attributes of the dataset, and the attributes of the dataset are selected based in part on the causal relationships.
7. The method as recited in claim 1 , further comprising performing a counterfactual analysis, and when the counterfactual analysis provides insight as to the amount of storage needed by the dataset, using the risk score and results of the counterfactual analysis to generate a recommendation.
8. The method as recited in claim 1 , wherein the shock information comprises a periodic change rate of consumption of storage space of the asset by the dataset.
9. The method as recited in claim 1 , wherein the drift information comprises information concerning unexpected changes in the dataset that affect an amount of storage consumed by the dataset.
10. The method as recited in claim 1 , wherein the drift information is derived based in part on space reclamation after a garbage collection process is performed with respect to the dataset, and based in part on a deduplication ratio of the dataset.
11. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
accessing a dataset;
selecting a list of attributes of the dataset, each of the attributes being selected based on a determination that the attribute is affecting growth of the dataset and affecting an amount of data storage space consumed by the dataset;
assigning a SHAP score to each attribute;
using the SHAP scores to assign respective weights to each attribute;
deriving drift and shock information for the dataset, and the drift and shock information is derived from the SHAP scores;
based on the drift and shock information, calculating a risk score that a storage capacity of an asset where the dataset is stored will be exhausted within a particular time interval; and
using the risk score as a basis to identify, and implement, an action to reduce a risk that the storage capacity of the asset will be exhausted within the particular time interval.
12. The non-transitory storage medium as recited in claim 11 , wherein the risk score comprises a probability that the storage capacity of the asset where the dataset is stored will be exhausted within the particular time interval.
13. The non-transitory storage medium as recited in claim 11 , wherein the drift is determined based on a weighted average of the attribute weights, and the shock is determined based on compression of data in the dataset.
14. The non-transitory storage medium as recited in claim 11 , wherein a size of the dataset changes nonlinearly over time.
15. The non-transitory storage medium as recited in claim 11 , wherein the risk score is calculated using Geometric Brownian Motion.
16. The non-transitory storage medium as recited in claim 11 , further comprising identifying one or more causal relationships among the attributes of the dataset, and the attributes of the dataset are selected based in part on the causal relationships.
17. The non-transitory storage medium as recited in claim 11 , further comprising performing a counterfactual analysis, and when the counterfactual analysis provides insight as to the amount of storage needed by the dataset, using the risk score and results of the counterfactual analysis to generate a recommendation.
18. The non-transitory storage medium as recited in claim 11 , wherein the shock information comprises a periodic change rate of consumption of storage space of the asset by the dataset.
19. The non-transitory storage medium as recited in claim 11 , wherein the drift information comprises information concerning unexpected changes in the dataset that affect an amount of storage consumed by the dataset.
20. The non-transitory storage medium as recited in claim 11 , wherein the drift information is derived based in part on space reclamation after a garbage collection process is performed with respect to the dataset, and based in part on a deduplication ratio of the dataset.