Method for minimizing storage requirements on fast/expensive arrays for data mobility and migration
Exemplary methods for data migration from a first storage system to a second storage system include maintaining a database containing deduplication information associated with a plurality of remote target storage systems. In one embodiment, the methods include identifying a plurality of storage devices of the first storage system as candidates to be migrated to the second storage system based on access patterns of the storage devices at the first storage system, and selecting one or more storage devices from the identified candidates to be migrated to a second one of the plurality of remote target storage systems that is selected based on the deduplication information of the database. In one embodiment, the methods further include migrating one or more datasets stored at the selected one or more storage devices to the second storage system.
1. A computer-implemented method for data migration from a first storage system to a second storage system, the method comprising:
maintaining a database containing deduplication information associated with a plurality of remote target storage systems, wherein the database is usable in determining through test deduplication a predicted deduplication ratio for any given dataset at each of the plurality of remote target storage systems, and wherein the deduplication information in the database is updated in response to migration of data to the plurality of remote target storage systems;
identifying a plurality of storage devices of the first storage system as candidates to be migrated to the second storage system based on access patterns of the storage devices at the first storage system, including identifying one or more candidate storage devices that have input and output (IO) usages greater than a predetermined threshold over a predetermined period of time;
for each of the candidate storage devices, performing a deduplication on data stored at the candidate storage device using the deduplication information of the database to determine a predicted deduplication ratio of the data for each of the plurality of remote target storage systems if the data was to be migrated to each of the plurality of remote target storage systems;
selecting one or more storage devices from the identified candidates to be migrated to one of the plurality of remote target storage systems as the second storage system that has a highest predicted deduplication ratio amongst the identified candidates; and
migrating one or more datasets stored at the selected one or more storage devices to the second storage system.
2. The method of claim 1 , wherein maintaining the database comprises:
selecting a first dataset that is stored at the first storage system;
generating deduplication information of the first dataset;
storing the generated deduplication information in the database; and
migrating the first dataset to the second storage system.
3. The method of claim 2 , wherein deduplication information includes fingerprints of data segments of the first dataset.
4. The method of claim 1 , wherein identifying the plurality of storage devices of the first storage system as candidates comprises:
for each storage device at the first storage system, determining an input/output (I/O) usage profile; and
selecting the plurality of storage devices with an I/O usage profile that satisfies a predetermined category.
5. The method of claim 1 , wherein a number of storage devices selected from the identified candidates is user configurable.
6. The method of claim 1 , further comprising updating the database to include deduplication information of the migrated one or more datasets.
7. A non-transitory computer-readable medium having computer instructions stored therein, which when executed by a processor, cause the processor to perform operations comprising:
maintaining a database containing deduplication information associated with a plurality of remote target storage systems, wherein the database is usable in determining through test deduplication a predicted deduplication ratio for any given dataset at each of the plurality of remote target storage systems, and wherein the deduplication information in the database is updated in response to migration of data to the plurality remote target storage systems;
identifying a plurality of storage devices of the first storage system as candidates to be migrated to the second storage system based on access patterns of the storage devices at the first storage system, including identifying one or more candidate storage devices that have input and output (IO) usages greater than a predetermined threshold over a predetermined period of time;
for each of the candidate storage devices, performing a deduplication on data stored at the candidate storage device using the deduplication information of the database to determine a predicted deduplication ratio of the data at each of the plurality of remote target storage system if the data was to be migrated to each of the plurality of remote target storage systems;
selecting one or more storage devices from the identified candidates to be migrated to one of the plurality of remote target storage systems as the second storage system that has a highest predicted deduplication ratio amongst the identified candidates; and
migrating one or more datasets stored at the selected one or more storage devices to the second storage system.
8. The non-transitory computer-readable medium of claim 7 , wherein maintaining the database comprises:
selecting a first dataset that is stored at the first storage system;
generating deduplication information of the first dataset;
storing the generated deduplication information in the database; and
migrating the first dataset to the second storage system.
9. The non-transitory computer-readable medium of claim 8 , wherein deduplication information includes fingerprints of data segments of the first dataset.
10. The non-transitory computer-readable medium of claim 7 , wherein identifying the plurality of storage devices of the first storage system as candidates comprises:
for each storage device at the first storage system, determining an input/output (I/O) usage profile; and
selecting the plurality of storage devices with an I/O usage profile that satisfies a predetermined category.
11. The non-transitory computer-readable medium of claim 7 , wherein a number of storage devices selected from the identified candidates is user configurable.
12. The non-transitory computer-readable medium of claim 7 , further comprising updating the database to include deduplication information of the migrated one or more datasets.
13. A data processing system, comprising:
a set of one or more processors; and
a set of one or more memories to store instructions, which when executed from the set of one or more memories, cause the set of one or more processors to
maintain a database containing deduplication information associated with a plurality of remote target storage systems, wherein the database is usable in determining through test deduplication a predicted deduplication ratio for any given dataset at each of the plurality of remote target storage systems, and wherein the deduplication information in the database is updated in response to migration of data to the plurality remote target storage systems;
identify a plurality of storage devices of the first storage system as candidates to be migrated to the second storage system based on access patterns of the storage devices at the first storage system, including identifying one or more candidate storage devices that have input and output (IO) usages greater than a predetermined threshold over a predetermined period of time,
for each of the candidate storage devices, perform a deduplication on data stored at the candidate storage device using the deduplication information of the database to determine a predicted deduplication ratio of the data for each of the plurality of remote target storage systems if the data was to be migrated to each of the plurality of remote target storage systems;
select one or more storage devices from the identified candidates to be migrated to one of the plurality of remote target storage systems as the second storage system that has a highest predicted deduplication ratio amongst the identified candidates, and
migrate one or more datasets stored at the selected one or more storage devices to the second storage system.
14. The data processing system of claim 13 , wherein maintaining the database comprises:
selecting a first dataset that is stored at the first storage system;
generating deduplication information of the first dataset;
storing the generated deduplication information in the database; and
migrating the first dataset to the second storage system.
15. The data processing system of claim 14 , wherein deduplication information includes fingerprints of data segments of the first dataset.
16. The data processing system of claim 13 , wherein identifying the plurality of storage devices of the first storage system as candidates comprises:
for each storage device at the first storage system, determining an input/output (I/O) usage profile; and
selecting the plurality of storage devices with an I/O usage profile that satisfies a predetermined category.
17. The data processing system of claim 13 , wherein a number of storage devices selected from the identified candidates is user configurable.
18. The data processing system of claim 13 , further comprising updating the database to include deduplication information of the migrated one or more datasets.