Method and system of large amount of data migration with enhanced efficiency
The present disclosure provides methods, systems, and non-transitory computer readable media for migrating data in a data storage system. A method for migrating data comprises: transferring data from a storage cluster in a first data center to a migration cartridge of the first data center; converting, by the migration cartridge of the first data center, the data's format into an archival format; storing the data into a media card; and moving the media card from the first data center to a second data center, wherein the data is read from the media card into a migration cartridge of the second data center and stored into an archival cluster in the second data center according to the archival format of the data.
1. A method for data migration, comprising:
transferring data from a storage cluster to a first migration cartridge;
converting, by the first migration cartridge, the data from a storage format used by the storage cluster into an archival format used by an archival cluster according to meta-data corresponding to the data, and converting the meta-data's format into an archival meta-data format, wherein the archival format is different than the storage format, wherein the converted data is stored on a NAND region corresponding to a zone in the archival cluster according to the archival format, and wherein the converted data is divided into multiple parts; and
storing the converted data and the meta-data in the archival meta-data format into a media card, wherein the converted data in the archival format is read sequentially from the media card into a second migration cartridge and stored sequentially into the archival cluster in the archival format directly, comprises:
storing each of the multiple parts of the converted data into respective NAND region of multiple NAND regions on the media card, wherein each of the multiple NAND regions on the media card correspond to a respective zone in the archival cluster; and
in response to the converted data being stored into the archival cluster in the archival format, the converted data is stored from each of the multiple NAND regions into the respective zone in the archival cluster according to the meta-data in the archival meta-data format.
2. The method of claim 1 , wherein converting the data's format into the archival format in the first migration cartridge further comprises:
encoding the converted data with an error correction code.
3. The method of claim 1 , wherein transferring data from the storage cluster to the first migration cartridge further comprises:
receiving, by the first migration cartridge, the meta-data corresponding to the data from a meta-data server;
locating the data in the storage cluster according to the meta-data; and
transferring the data from the storage cluster to the first migration cartridge according to the location of the data in the storage cluster.
4. The method of claim 3 , wherein the archival meta-data format includes meta-data for error correction code parity, the method further comprising:
storing the meta-data into a meta-data drive in the first migration cartridge; and
storing the meta-data in the archival meta-data format into the media card.
5. The method of claim 1 , wherein transferring data from the storage cluster to the first migration cartridge further comprises:
transferring the data from multiple copies of the data stored in the storage cluster in parallel.
6. The method of claim 1 , further comprising:
moving the media card from the first migration cartridge to the second migration cartridge, wherein while the media card is moved from the first migration cartridge to the second migration cartridge, the data stored in the storage cluster is deleted or overwritten.
7. The method of claim 1 , wherein the media card comprises one or more NAND switches that fan out to multiple NAND regions.
8. A non-transitory computer readable medium that stores a set of instructions that is executable by at least one processor of a computer system to cause the computer system to perform a method, the method comprising:
transferring data from a storage cluster to a first migration cartridge;
converting, by the first migration cartridge, the data from a storage format used by the storage cluster into an archival format used by an archival cluster according to meta-data corresponding to the data, and converting the meta-data's format into an archival meta-data format, wherein the archival format is different than the storage format, wherein the converted data is stored on a NAND region corresponding to a zone in the archival cluster according to the archival format, and wherein the converted data is divided into multiple parts; and
storing the converted data and the meta-data in the archival meta-data format into a media card, wherein the converted data in the archival format is read sequentially from the media card into a second migration cartridge and stored sequentially into the archival cluster in the archival format directly, comprises:
storing each of the multiple parts of the converted data into respective NAND region of multiple NAND regions on the media card, wherein:
each of the multiple NAND regions on the media card correspond to a respective zone in the archival cluster, and
in response to the converted data being stored into the archival cluster in the archival format, the converted data is stored from each of the multiple NAND regions into the respective zone in the archival cluster according to the meta-data in the archival meta-data format.
9. The non-transitory computer readable medium of claim 8 , wherein the set of instructions is executable by the at least one processor of the computer system to cause the computer system to further perform:
encoding the converted data with an error correction code.
10. The non-transitory computer readable medium of claim 8 , wherein the set of instructions is executable by the at least one processor of the computer system to cause the computer system to further perform:
receiving, by the first migration cartridge, the meta-data corresponding to the data from a meta-data server;
locating the data in the storage cluster according to the meta-data; and
transferring the data from the storage cluster to the first migration cartridge according to the location of the data in the storage cluster.
11. The non-transitory computer readable medium of claim 10 , wherein the archival meta-data format includes meta-data for error correction code parity, and the set of instructions is executable by the at least one processor of the computer system to cause the computer system to further perform:
storing the meta-data into a meta-data drive in the first migration cartridge; and
storing the meta-data in the archival meta-data format into the media card.
12. The non-transitory computer readable medium of claim 8 , wherein while the media card is moved from the first migration cartridge to the second migration cartridge, the data stored in the storage cluster is deleted or overwritten.
13. A system for data migration, comprising:
a storage cluster;
a first migration cartridge communicatively coupled to the storage cluster, wherein the first migration cartridge comprises one or more slots for inserting media cards;
a memory storing a set of instructions; and
one or more processors configured to execute the set of instructions to cause the system to perform:
transferring data from the storage cluster to the first migration cartridge;
converting, by the first migration cartridge, the data from a storage format used by the storage cluster into an archival format used by an archival cluster according to meta-data corresponding to the data, and converting the meta-data's format into an archival meta-data format, wherein the archival format is different than the storage format, wherein the converted data is stored on a NAND region corresponding to a zone in the archival cluster according to the archival format, and wherein the converted data is divided into multiple parts; and
storing the converted data and the meta-data in the archival meta-data format into a media card, wherein the converted data in the archival format is read sequentially from the media card into a second migration cartridge and stored sequentially into the archival cluster in the archival format directly, comprises:
storing each of the multiple parts of the converted data into respective NAND region of multiple NAND regions on the media card, wherein:
each of the multiple NAND regions on the media card correspond to a respective zone in the archival cluster, and
in response to the converted data being stored into the archival cluster in the archival format, the converted data is stored from each of the multiple NAND regions into respective zone in the archival cluster according to the meta-data in the archival meta-data format.
14. The system of claim 13 , wherein the one or more processors are further configured to execute the set of instructions to cause the system to perform:
encoding the converted data with an error correction code.
15. The system of claim 13 , wherein the one or more processors are further configured to execute the set of instructions to cause the system to perform:
receiving, by the first migration cartridge, the meta-data corresponding to the data from a meta-data server;
locating the data in the storage cluster according to the meta-data; and
transferring the data from the storage cluster to the first migration cartridge according to the location of the data in the storage cluster.
16. The system of claim 15 , wherein the archival meta-data format includes meta-data for error correction code parity, and the one or more processors are further configured to execute the set of instructions to cause the system to perform:
storing the meta-data into a meta-data drive in the first migration cartridge; and
storing the meta-data in the archival meta-data format into the media card.
17. The system of claim 13 , wherein while the media card is moved from the first migration cartridge to the second migration cartridge, the data stored in the storage cluster is deleted or overwritten.