Data migration method, system and device for storage server, and non-transitory readable storage medium
Disclosed are a data migration method, system and device for a storage server, and a non-transitory readable storage medium. The method includes: according to storage information of a storage server, determining the number n of strips to be migrated; selecting n stripes to be migrated from among c full stripes according to a preset migration rule, and selecting, from the n stripes to be migrated, data blocks to be migrated and migrating the data blocks to the newly inserted disks, so that after the migration is completed, the difference in stripe usage between any two disks in the storage server does not exceed 1; and determining data content of check data blocks of the strips so as to complete data migration of the storage server, and determining data content of check data blocks of the c-n strips by means of incremental update of an erasure code.
1 . A data migration method for a storage server, comprising:
determining storage information of a storage server, wherein, the storage information is storage information of the storage server before inserting m disks;
determining, according to the storage information, the number n of strips to be migrated, wherein, the strips to be migrated are used to migrate to the m disks which are inserted into the storage server;
selecting n strips from among c strips according to a preset migration rule to serve as the strips to be migrated, and selecting, from the n strips to be migrated, data blocks to be migrated and migrating the data blocks from an original disk where the data blocks to be migrated is located to the disks which are newly inserted, so that after migration of the data blocks is completed, difference in strip usage between any two disks of the m disks in the storage server does not exceed 1, wherein the preset migration rule comprises: for any one of the data blocks to be migrated, determining a strip in which a check data block with the lowest strip serial number is located in a disk where the data block to be migrated is located before migration, and when the strip has a storage space that is not occupied by any data block to be migrated in the m disks which are newly inserted, migrating the data block to be migrated to the storage space, thereby a position limitation is imposed on placement of the data blocks to be migrated into the m newly inserted disks, so as to satisfy the requirements of load balancing; and
determining data content of check data blocks of the strips so as to complete data migration of the storage server, and for c-n strips among the c strips that are not selected as the strips to be migrated, determining data content of check data blocks of the c-n strips by means of incremental update of an erasure code, user data blocks of the c-n strips which are remaining in original disks are not moved, so as to perform incremental update of the erasure code by combining data migrated to the m newly inserted disks,
a type of the erasure code is Reed-Solomon Code (RS code) used for encoding k data blocks into r additional check blocks, parameters of the RS code, when implementing a storage erasure coding function, are determined on the basis of location information, a general algorithm relation of the erasure codes is summarized as:
P
1
=
a
1
,
1
D
1
⊕
a
1
,
2
D
2
⊕
…
⊕
a
1
,
k
D
k
…
P
r
=
a
r
,
1
D
1
⊕
a
r
,
2
D
2
⊕
…
⊕
a
r
,
k
D
k
,
wherein, a 1,1 to a r,k are selected from an encoding matrix of a Vandermonde matrix or an encoding matrix of a Cauchy matrix, P 1 to Pr are check data blocks, D 1 to Dk are user data blocks, ⊕ is addition in the Galois field,
determining whether a total number of user data blocks before the m disks are newly inserted into the storage server is consistent with a total number of user data blocks after the m disks are inserted and data migration is performed, and in a case that the total number of user data blocks before the m disks are newly inserted into the storage server is inconsistent with the total number of user data blocks after the m disks are inserted and data migration is performed, determining a part of the user data blocks is not successfully migrated to the newly inserted disks,
wherein n<c, m, n and c are all positive integers, and c represents the number of full strips in use by of the storage server before the m disks are newly inserted into the storage server,
wherein selecting the n stripes from among the c stripes according to the preset migration rule to serve as the stripes to be migrated comprises:
using the n strips with highest strip serial numbers among the c strips to serve as the strips to be migrated,
wherein the preset migration rule further comprises:
according to a rule that a stripe with a high stripe serial number takes precedence over a stripe with a low stripe serial number and when stripe serial numbers of two stripes are the same, a disk with a high disk serial number to which a data block to be migrated belongs takes precedence over a disk with a low disk serial number to which a data block to be migrated belongs, successively selecting each data block to be migrated from the n stripes to be migrated, and migrating same to the disks which are newly inserted; and
upon completion of migration of any one of the data blocks to be migrated, in a case that the difference in the stripe usage between the any two disks in the storage server does not exceed 1, determining the migration being completed.
2 . The data migration method for the storage server according to claim 1 , wherein determining the storage information of the storage server comprises:
determining a number r of disks, the number c of the full strips in use by the storage server, and a number p of check data blocks of a single strip before the m disks are newly inserted into the storage server.
3 . The data migration method for the storage server according to claim 1 , wherein determining the storage information of the storage server comprises:
determining a current disk space usage of the storage server and whether storage degradation occurs in the storage server.
4 . The data migration method for the storage server according to claim 1 , wherein determining the storage information of the storage server further comprises:
updating the storage information of the storage server in real time or periodically.
5 . The data migration method for the storage server according to claim 2 , wherein determining, according to the storage information, the number n of the strips to be migrated comprises:
determining whether the following formula holds or not:
(
c
*
m
r
-
p
+
m
-
⌊
c
*
m
r
-
p
+
m
⌋
)
*
(
r
-
p
)
>
m
;
in a case that the following formula holds, the number of the strips to be migrated which is determined being:
n
=
⌊
c
*
m
r
-
p
+
m
⌋
+
1
;
in a case that the following formula not holds, the number of the strips to be migrated which is determined being:
n
=
⌊
c
*
m
r
-
p
+
m
⌋
.
6 . The data migration method for the storage server according to claim 1 , wherein after determining the storage information of the storage server, the method further comprises:
determining whether a strip that is not fully written exists before the m disks are newly inserted into the storage server;
in a case that the strip that is not fully written does not exist, executing the operation of determining, according to the storage information, the number n of the strips to be migrated; and
in a case that the strip that is not fully written exists, considering the strip that is not fully written to be an empty strip, and executing the operation of determining, according to the storage information, the number n of the strips to be migrated.
7 . The data migration method for the storage server according to claim 1 , wherein after determining the data content of the check data blocks of the stripes, the method further comprises:
verifying whether a data migration process of the storage server is anomalous, and in a case that the data migration process of the storage server is anomalous, outputting prompt information.
8 . The data migration method for the storage server according to claim 7 , wherein after determining that the data migration process of the storage server is anomalous, the method further comprises:
recording an event log of this data migration process.
9 . The data migration method for the storage server according to claim 1 , wherein the storage server is a storage server in a distributed storage system.
10 . The data migration method for the storage server according to claim 1 , wherein the storage server is a storage server in a unified storage system.
11 . The data migration method for the storage server according to claim 1 , wherein the check data blocks are arranged in a left-handed misalignment manner.
12 . The data migration method for the storage server according to claim 1 , wherein the same encoding matrix is used to calculate erasure codes of the strips.
13 . A data migration device of a storage server, comprising:
a memory, configured to store a computer program; and
a processor, configured to execute the computer program to:
determine storage information of a storage server, wherein, the storage information is storage information of the storage server before inserting m disks;
determine, according to the storage information, the number n of strips to be migrated, wherein, the strips to be migrated are used to migrate to the m disks which are inserted into the storage server;
select n strips from among c strips according to a preset migration rule to serve as the strips to be migrated, and selecting, from the n strips to be migrated, data blocks to be migrated and migrating the data blocks from an original disk where the data blocks to be migrated is located to the disks which are newly inserted, so that after migration of the data blocks is completed, difference in strip usage between any two disks of the m disks in the storage server does not exceed 1, wherein the preset migration rule comprises: for any one of the data blocks to be migrated, determining a strip in which a check data block with the lowest strip serial number is located in a disk where the data block to be migrated is located before migration, and when the strip has a storage space that is not occupied by any data block to be migrated in the m disks which are newly inserted, migrating the data block to be migrated to the storage space, thereby a position limitation is imposed on placement of the data blocks to be migrated into the m newly inserted disks, so as to satisfy the requirements of load balancing; and
determine data content of check data blocks of the strips so as to complete data migration of the storage server, and for c-n strips among the c strips that are not selected as the strips to be migrated, determine data content of check data blocks of the c-n strips by means of incremental update of an erasure code, user data blocks of the c-n strips which are remaining in original disks are not moved, so as to perform incremental update of the erasure code by combining data migrated to the m newly inserted disks,
a type of the erasure code is Reed-Solomon Code (RS code) used for encoding k data blocks into r additional check blocks, parameters of the RS code, when implementing a storage erasure coding function, are determined on the basis of location information, a general algorithm relation of the erasure codes is summarized as:
P
1
=
a
1
,
1
D
1
⊕
a
1
,
2
D
2
⊕
…
⊕
a
1
,
k
D
k
…
P
r
=
a
r
,
1
D
1
⊕
a
r
,
2
D
2
⊕
…
⊕
a
r
,
k
D
k
,
wherein, a 1,1 to a r,k are selected from an encoding matrix of a Vandermonde matrix or an encoding matrix of a Cauchy matrix, P 1 to Pr are check data blocks, D 1 to Dk are user data blocks, ⊕ is addition in the Galois field,
determine whether a total number of user data blocks before the m disks are newly inserted into the storage server is consistent with a total number of user data blocks after the m disks are inserted and data migration is performed, and in a case that the total number of user data blocks before the m disks are newly inserted into the storage server is inconsistent with the total number of user data blocks after the m disks are inserted and data migration is performed, determine a part of the user data blocks is not successfully migrated to the newly inserted disks,
wherein n<c, m, n and c are all positive integers, and c represents the number of full strips in use by of the storage server before the m disks are newly inserted into the storage server,
wherein select the n stripes from among the c stripes according to the preset migration rule to serve as the stripes to be migrated comprises:
use the n strips with highest strip serial numbers among the c strips to serve as the strips to be migrated,
wherein the preset migration rule further comprises:
according to a rule that a stripe with a high stripe serial number takes precedence over a stripe with a low stripe serial number and when stripe serial numbers of two stripes are the same, a disk with a high disk serial number to which a data block to be migrated belongs takes precedence over a disk with a low disk serial number to which a data block to be migrated belongs, successively select each data block to be migrated from the n stripes to be migrated, and migrate same to the disks which are newly inserted; and
upon completion of migration of any one of the data blocks to be migrated, in a case that the difference in the stripe usage between the any two disks in the storage server does not exceed 1, determine the migration being completed.
14 . A non-transitory readable storage medium, wherein the non-transitory readable storage medium stores a computer program which, when being executed by a processor, cause the processor to:
determine storage information of a storage server, wherein, the storage information is storage information of the storage server before inserting m disks;
determine, according to the storage information, the number n of strips to be migrated, wherein, the strips to be migrated are used to migrate to the m disks which are inserted into the storage server;
select n strips from among c strips according to a preset migration rule to serve as the strips to be migrated, and selecting, from the n strips to be migrated, data blocks to be migrated and migrating the data blocks from an original disk where the data blocks to be migrated is located to the disks which are newly inserted, so that after migration of the data blocks is completed, difference in strip usage between any two disks of the m disks in the storage server does not exceed 1, wherein the preset migration rule comprises: for any one of the data blocks to be migrated, determining a strip in which a check data block with the lowest strip serial number is located in a disk where the data block to be migrated is located before migration, and when the strip has a storage space that is not occupied by any data block to be migrated in the m disks which are newly inserted, migrating the data block to be migrated to the storage space, thereby a position limitation is imposed on placement of the data blocks to be migrated into the m newly inserted disks, so as to satisfy the requirements of load balancing; and
determine data content of check data blocks of the strips so as to complete data migration of the storage server, and for c-n strips among the c strips that are not selected as the strips to be migrated, determine data content of check data blocks of the c-n strips by means of incremental update of an erasure code, user data blocks of the c-n strips which are remaining in original disks are not moved, so as to perform incremental update of the erasure code by combining data migrated to the m newly inserted disks,
a type of the erasure code is Reed-Solomon Code (RS code) used for encoding k data blocks into r additional check blocks, parameters of the RS code, when implementing a storage erasure coding function, are determined on the basis of location information, a general algorithm relation of the erasure codes is summarized as:
P
1
=
a
1
,
1
D
1
⊕
a
1
,
2
D
2
⊕
…
⊕
a
1
,
k
D
k
…
P
r
=
a
r
,
1
D
1
⊕
a
r
,
2
D
2
⊕
…
⊕
a
r
,
k
D
k
,
wherein, a 1,1 to a r,k are selected from an encoding matrix of a Vandermonde matrix or an encoding matrix of a Cauchy matrix, P 1 to Pr are check data blocks, D 1 to Dk are user data blocks, ⊕ is addition in the Galois field,
determine whether a total number of user data blocks before the m disks are newly inserted into the storage server is consistent with a total number of user data blocks after the m disks are inserted and data migration is performed, and in a case that the total number of user data blocks before the m disks are newly inserted into the storage server is inconsistent with the total number of user data blocks after the m disks are inserted and data migration is performed, determine a part of the user data blocks is not successfully migrated to the newly inserted disks,
wherein n<c, m, n and c are all positive integers, and c represents the number of full strips in use by of the storage server before the m disks are newly inserted into the storage server,
wherein select the n stripes from among the c stripes according to the preset migration rule to serve as the stripes to be migrated comprises:
use the n strips with highest strip serial numbers among the c strips to serve as the strips to be migrated,
wherein the preset migration rule further comprises:
according to a rule that a stripe with a high stripe serial number takes precedence over a stripe with a low stripe serial number and when stripe serial numbers of two stripes are the same, a disk with a high disk serial number to which a data block to be migrated belongs takes precedence over a disk with a low disk serial number to which a data block to be migrated belongs, successively select each data block to be migrated from the n stripes to be migrated, and migrate same to the disks which are newly inserted; and
upon completion of migration of any one of the data blocks to be migrated, in a case that the difference in the stripe usage between the any two disks in the storage server does not exceed 1, determine the migration being completed.