Data block distribution in hybrid storage systems

1 . A method for data processing implemented on a computing device having at least one processor and at least one storage device, comprising:

obtaining information of one or more hard disks of each of a plurality of data nodes, the information of the one or more hard disks at least including type information, capacity information, and location information of each of the one or more hard disks;

determining, based on a data processing instruction, a data block to be stored, the data processing instruction being received from a metadata server;

determining a data file type of the data block to be stored;

determining target information of a target hard disk among the one or more hard disks based on the data file type of the data block to be stored, the information of the one or more hard disks, and a priority strategy; and

transmitting the data processing instruction including the target information of the target hard disk to a target data node corresponding to the target hard disk, wherein the priority strategy includes a node allocating strategy and a hard disk issuing strategy, and the node allocating strategy includes:

determining one or more points on a load ring corresponding to each of the plurality of data nodes and a point on the load ring corresponding to the data block to be stored; and

determining the target data node based on the corresponding points on the load ring according to a preset rule, wherein

the corresponding points of the data nodes on the load ring are determined by performing a hash operation on features of the data nodes, and

the preset rule includes starting from the corresponding point of the data block to be stored on the load ring, traversing the points on the load ring in a clockwise or counterclockwise order, and determining a data node corresponding to the corresponding point traversed at first on the load ring as the target data node, and the hard disk issuing strategy includes:

determining a priority value of each of the one or more hard disks in the target data node, the priority value of the hard disk indicating a priority degree of storing the data block in the hard disk; and

determining a hard disk with a maximum priority value as the target hard disk and the corresponding location information of the target hard disk, wherein the priority value of the hard disk is related to a data increment of the hard disk, the data increment of the hard disk refers to an amount of data newly stored in the hard disk within a preset time period, and the smaller the data increment of the hard disk is, the larger the priority value of the hard disk is, and

the determining a priority value of each of the one or more hard disks in the target data node includes:

obtaining the priority value of the hard disk by weighting, based on a weight vector, an available capacity of the hard disk, a read and write load of the hard disk, the data increment of the hard disk, and a fragmentation rate of the hard disk, wherein

the weight vector includes a weight of the available capacity of the hard disk, a weight of the read and write load of the hard disk, a weight of the data increment of the hard disk, and a weight of the fragmentation rate of the hard disk.

2 . The method of claim 1 , wherein the determining target information of a target hard disk among the one or more hard disks based on the data file type of the data block to be stored, the information of the one or more hard disks, and a priority strategy further includes:

in response to determining that the data file type of the data block to be stored is a first data file type, determining, based on the priority strategy, a first hard disk as the target hard disk; or

in response to determining that the data file type of the data block to be stored is a second data file type, determining, based on the priority strategy, a second hard disk as the target hard disk, wherein the priority strategy further includes:

determining whether capacity information of the first hard disk whose type corresponds to the data file type of the data block satisfies a condition;

in response to determining that the capacity information of the first hard disk satisfies the condition, designating the first hard disk as the target hard disk; or

in response to determining that the capacity information of the first hard disk whose type corresponds to the data file type of the data block does not satisfy the condition, designating the second hard disk whose type does not correspond to the data file type as the target hard disk.

3 . The method of claim 2 , wherein

a life cycle of the data block in the first data file type is not fixed and capacity occupation of the data block in the first data file type does not exceed a threshold,

a life cycle of the data block in the second data file type is fixed and capacity occupation of the data block in the second data file type exceeds the threshold,

the first hard disk is a conventional magnetic recording (CMR) hard disk, and

the second hard disk is a shingled magnetic recording (SMR) hard disk.

4 . The method of claim 1 , wherein the data processing instruction includes a data storage instruction, and

the method further includes:

determining whether the data block to be stored is stored in the target hard disk; and

in response to determining that the data block to be stored is not stored in the target hard disk, updating the data storage instruction by replacing the target hard disk with another hard disk whose type does not correspond to the data file type.

5 . The method of claim 4 , further comprising:

in response to determining that the data block to be stored is stored in a hard disk, obtaining location information of the hard disk where the data block is stored; and

establishing, based on the location information of the hard disk corresponding to the data block, a relationship table including a name of a data file, a name of the data block, and the location information of the hard disk.

6 . A system for data processing, comprising:

at least one storage device including a set of instructions; and

at least one processor in communication with the at least one storage device, wherein when executing the set of instructions, the at least one processor is directed to perform operations including:

obtaining information of one or more hard disks of each of a plurality of data nodes, the information of the one or more hard disks at least including type information, capacity information, and location information of each of the one or more hard disks;

determining, based on a data processing instruction, a data block to be stored, the data processing instruction being received from a metadata server;

determining a data file type of the data block to be stored;

determining target information of a target hard disk among the one or more hard disks based on the data file type of the data block to be stored, the information of the one or more hard disks, and a priority strategy; and

transmitting the data processing instruction including the target information of the target hard disk to a target data node corresponding to the target hard disk, wherein the priority strategy includes a hard disk issuing strategy, and the hard disk issuing strategy includes:

determining a priority value of each of the one or more hard disks in the target data node, the priority value of the hard disk indicating a priority degree of storing the data block in the hard disk; and

determining a hard disk with a maximum priority value as the target hard disk and the corresponding location information of the target hard disk, wherein the priority value of the hard disk is related to a data increment of the hard disk, the data increment of the hard disk refers to an amount of data newly stored in the hard disk within a preset time period, and the smaller the data increment of the hard disk is, the larger the priority value of the hard disk is, and

the determining a priority value of each of the one or more hard disks in the target data node includes:

obtaining the priority value of the hard disk by weighting, based on a weight vector, an available capacity of the hard disk, a read and write load of the hard disk, the data increment of the hard disk, and the fragmentation rate of the hard disk, wherein

the weight vector includes a weight of the available capacity of the hard disk, a weight of the read and write load of the hard disk, a weight of the data increment of the hard disk, and a weight of the fragmentation rate of the hard disk.

7 . A method for data processing implemented on a computing device having at least one processor and at least one storage device, comprising:

receiving a data processing instruction including target information to a target data node corresponding to a target hard disk; and

determining, based on the data processing instruction, the target hard disk in the target data node, wherein the target information is determined by:

obtaining information of one or more hard disks of each of a plurality of data nodes, the information of the one or more hard disks at least including type information, capacity information, and location information of each of the one or more hard disks;

determining, based on a data processing instruction, a data block to be stored, the data processing instruction being received from a metadata server;

determining a data file type of the data block to be stored; and

determining target information of the target hard disk among the one or more hard disks based on the data file type of the data block to be stored, the information of the one or more hard disks, and a priority strategy, wherein the priority strategy includes a hard disk issuing strategy, and the hard disk issuing strategy includes:

determining a priority value of each of the one or more hard disks in the target data node, the priority value of the hard disk indicating a priority degree of storing the data block in the hard disk; and

determining a hard disk with a maximum priority value as the target hard disk and the corresponding location information of the target hard disk, wherein the priority value of the hard disk is related to a data increment of the hard disk, the data increment of the hard disk refers to an amount of data newly stored in the hard disk within a preset time period, and the smaller the data increment of the hard disk is, the larger the priority value of the hard disk is, and

the determining a priority value of each of the one or more hard disks in the target data node includes:

obtaining the priority value of the hard disk by weighting, based on a weight vector, an available capacity of the hard disk, a read and write load of the hard disk, the data increment of the hard disk, and the fragmentation rate of the hard disk, wherein

the weight vector includes a weight of the available capacity of the hard disk, a weight of the read and write load of the hard disk, a weight of the data increment of the hard disk, and a weight of the fragmentation rate of the hard disk.

8 . The method of claim 7 , wherein the receiving a data processing instruction including target information to a target data node corresponding to a target hard disk includes:

determining the information of the one or more hard disks of each of a plurality of data nodes, the information of the one or more hard disks at least including the type information, the capacity information, and the location information of each of the one or more hard disks;

transmitting the information of the one or more hard disks of each of the one or more data nodes to a metadata server; and

receiving the data processing instruction including the target information to the target data node corresponding to the target hard disk from the metadata server.

9 . The method of claim 7 , wherein before the determining, based on the data processing instruction, the target hard disk in the target data node, the method includes:

determining, based on the data processing instruction, target type information of the target hard disk; and

determining, based on the target type information and a relationship between type information of hard disks and service units, a target service unit corresponding to the target type information.

10 . The method of claim 9 , further comprising:

obtaining the data block to be stored; and

processing, based on the data processing instruction, the data block to be stored by retrieving the target service unit corresponding to the target type information.

11 . The method of claim 9 , wherein the data processing instruction includes a data storage instruction, and

the method further includes:

determining, based on the data storage instruction, the target hard disk where the data block in the data storage instruction is stored; and

determining, based on the target type information of the target hard disk, the target service unit to store the data block.

12 . The method of claim 11 , further comprising:

determining whether the data block to be stored is stored in the target hard disk; and

in response to determining that the data block to be stored is not stored in the target hard disk, obtaining an updated data storage instruction including replacing the target hard disk with another hard disk whose type does not correspond to a data file type of the data block.

13 . The method of claim 12 , further comprising:

in response to determining that the data block to be stored is stored in a hard disk, obtaining location information of the hard disk; and

transmitting the location information of the hard disk to the metadata server.

14 . The method of claim 10 , wherein the determining, based on the target type information and a relationship between type information of hard disks and service units, a target service unit corresponding to the target type information includes:

in response to determining that the target type information is a conventional magnetic recording (CMR) hard disk, determining a service unit of the CMR hard disk to be associated with the target hard disk; or

in response to determining that the target type information is a shingled magnetic recording (SMR) hard disk, determining a service unit of the SMR hard disk to be associated with the target hard disk.

15 . The method of claim 1 , wherein in response to the type of the data block to be stored as a first data file type, the priority value of the hard disk is related to a fragmentation rate of the hard disk, the fragmentation rate of the hard disk referring to a proportion of file fragmentation in the hard disk.

16 . The method of claim 1 , wherein the weight vector is determined based on a weight determination model, wherein an input of the weight determination model includes scale information and load information, and an output of the weight determination model includes the weight vector.

17 . The method of claim 16 , wherein the scale information refers to information related to a scale of a distributed storage cluster including the plurality of data nodes, and the scale information includes a number of data nodes of the distributed storage cluster, a total number of hard disks of the distributed storage cluster, and a total capacity of the hard disks of the distributed storage cluster.

18 . The method of claim 16 , wherein the weight determination model is obtained by training an initial weight determination model using training samples and corresponding labels, wherein the training samples include historical scale information and historical load information, and the labels include weight vectors corresponding to the historical scale information and historical load information.

19 . The method of claim 18 , wherein

the labels of the training samples are determined based on an operation stability and an operation efficiency of a distributed storage cluster including the plurality of data nodes, and

a weight vector corresponding to a training sample with high operation stability and high operation efficiency is determined as the label of the training sample.

20 . The system of claim 6 , wherein the priority strategy further includes a node allocating strategy a hard disk issuing strategy, and the node allocating strategy includes:

determining one or more points on a load ring corresponding to each of the plurality of data nodes and a point on the load ring corresponding to the data block to be stored; and

determining the target data node based on the corresponding points on the load ring according to a preset rule, wherein

the corresponding points of the data nodes on the load ring are determined by performing a hash operation on features of the data nodes, and

the preset rule includes starting from the corresponding point of the data block to be stored on the load ring, traversing the points on the load ring in a clockwise or counterclockwise order, and determining a data node corresponding to the corresponding point traversed at first on the load ring as the target data node.