Processes and methods of aggregating, analyzing, and reporting distributed storage system properties within a central platform using storage descriptor data structures
Example implementations described herein are directed to a storage descriptor data structure that can represent characteristics of the dataset stored in an underlying volume without requiring the dataset in the volume to be migrated for processing. Such data structures are more compressed than the original data structure and can therefore facilitate storage allocation before the migration is conducted.
1. A method, comprising:
determining volume identification information and a probabilistic data structure indicative of size of duplicate data in a volume from sampling data stored in the volume;
generating storage descriptor data describing the volume from the volume identification information and the probabilistic data structure;
storing the storage descriptor data in a centralized management system;
generating a user interface configured to indicate a state of data reduction of the volume;
receiving a selection of a plurality of volumes managed by the centralized management system; and
combining the storage descriptor data associated with the plurality of volumes to indicate the state of data reduction of the plurality of volumes;
wherein combining the storage descriptor data comprises combining one or more of hyperloglog information, compression information, and number of zero pages from the storage descriptor data of each of the plurality of volumes.
2. The method of claim 1 , wherein the storage descriptor data comprises an expected physical capacity required for allocation in a storage array for conducting migration on the volume.
3. The method of claim 1 , wherein combining the storage descriptor data comprises combining one or more of a probabilistic data structure and compression information from the storage descriptor data of each of the plurality of volumes.
4. A method, comprising:
determining volume identification information and a probabilistic data structure indicative of size of duplicate data in a volume from sampling data stored in the volume;
generating storage descriptor data describing the volume from the volume identification information and the probabilistic data structure;
storing the storage descriptor data in a centralized management system;
generating a user interface configured to indicate a state of data reduction of the volume;
receiving a selection of a plurality of volumes managed by the centralized management system; and
combining the storage descriptor data associated with the plurality of volumes to indicate the state of data reduction of the plurality of volumes;
wherein combining the storage descriptor data comprises combining a data reduction rate from the storage descriptor data of each of the plurality of volumes.
5. The method of claim 1 , wherein the method is executed in response to a migration of the volume to a storage system.
6. The method of claim 1 , wherein the method is executed by a storage system managing the volume.
7. The method of claim 1 , wherein the centralized management system manages the storage descriptor data for a plurality of volumes and sub-aggregates the plurality of volumes based on the volume identification information in the storage descriptor data of each of the plurality of volumes.
8. A non-transitory computer readable medium, storing instructions for executing a process, the instructions comprising:
determining volume identification information and a probabilistic data structure indicative of size of duplicate data in a volume from sampling data stored in the volume;
generating storage descriptor data describing the volume from the volume identification information and the probabilistic data structure;
storing the storage descriptor data in a centralized management system;
generating a user interface configured to indicate a state of data reduction of the volume;
receiving a selection of a plurality of volumes managed by the centralized management system; and
combining the storage descriptor data associated with the plurality of volumes to indicate the state of data reduction of the plurality of volumes;
wherein combining the storage descriptor data comprises combining one or more of hyperloglog information, compression information, and number of zero pages from the storage descriptor data of each of the plurality of volumes.
9. The non-transitory computer readable medium of claim 8 , wherein the storage descriptor data comprises an expected physical capacity required for allocation in a storage array for conducting migration on the volume.
10. The non-transitory computer readable medium of claim 8 , wherein combining the storage descriptor data comprises combining one or more of a probabilistic data structure and compression information from the storage descriptor data of each of the plurality of volumes.
11. A non-transitory computer readable medium, storing instructions for executing a process, the instructions comprising:
determining volume identification information and a probabilistic data structure indicative of size of duplicate data in a volume from sampling data stored in the volume;
generating storage descriptor data describing the volume from the volume identification information and the probabilistic data structure;
storing the storage descriptor data in a centralized management system;
generating a user interface configured to indicate a state of data reduction of the volume;
receiving a selection of a plurality of volumes managed by the centralized management system; and
combining the storage descriptor data associated with the plurality of volumes to indicate the state of data reduction of the plurality of volumes;
wherein combining the storage descriptor data comprises combining a data reduction rate from the storage descriptor data of each of the plurality of volumes.
12. The non-transitory computer readable medium of claim 8 , wherein instructions are executed in response to a migration of the volume to a storage system.
13. The non-transitory computer readable medium of claim 8 , wherein the instructions are executed by a storage system managing the volume.
14. The non-transitory computer readable medium of claim 8 , wherein the centralized management system manages the storage descriptor data for a plurality of volumes and sub-aggregates the plurality of volumes based on the volume identification information in the storage descriptor data of each of the plurality of volumes.
15. A storage system configured to manage a plurality of volumes, the storage system comprising:
a processor, configured to:
determine volume identification information and a probabilistic data structure indicative of size of duplicate data in a volume from the plurality of volumes from sampling data stored in the volume;
generate storage descriptor data describing the volume from the volume identification information and the probabilistic data structure;
store the storage descriptor data in a centralized management system;
generate a user interface configured to indicate a state of data reduction of the volume;
receive a selection of a plurality of volumes managed by the centralized management system; and
combine the storage descriptor data associated with the plurality of volumes to indicate the state of data reduction of the plurality of volumes;
wherein the processor is configured to combine the storage descriptor data by combining one or more of hyperloglog information, compression information, and number of zero pages from the storage descriptor data of each of the plurality of volumes.