Data write-in method and apparatus in a distributed file system
Embodiments of the disclosure provide a data write-in method and apparatus. The data write-in method includes: selecting a target replica server from a plurality of replica servers managed by a metadata server according to remaining capacity of the plurality of replica servers; selecting a write-in disk in the target replica server according to remaining capacity and load of disks managed by the target replica server; and storing write-in data into the write-in disk through the target replica server.
1. A data write-in method, comprising:
selecting at least one target replica server from a plurality of replica servers managed by a metadata server according to remaining capacity of the plurality of replica servers;
selecting a write-in disk among a plurality of disks in the selected target replica server according to remaining capacity and load of each of the plurality of disks managed by the target replica server, wherein selecting the write-in disk among the plurality of disks in the target replica server further comprises:
determining a probability of a disk among the plurality of disks being selected according to the remaining capacity and the load of each of the plurality of disks managed by the target replica server; and
determining the write-in disk according to the probabilities of the plurality of disks; and
storing write-in data into the write-in disk through the target replica server.
2. The method according to claim 1 , wherein determining the probability of the disk among the plurality of disks being selected further comprises:
determining a score of each of the plurality of disks according to a proportion of the remaining capacity to a total capacity of each of the plurality of disks, a weight of the remaining capacity of each of the plurality of disks, the load of each of the plurality of disks, and a weight of the load of each of the plurality of disks; and
normalizing the score of each of the plurality of disks to obtain the probability of each of the plurality of disks being selected.
3. The method according to claim 2 , wherein determining the score of each of the plurality of disks further comprises:
determining a score of the proportion of the remaining capacity to the total capacity of each of the plurality of disks, and a score of the load of each of the plurality of disks, wherein the relationship between the score of the proportion and the proportion is expressed as a first monotonically increasing function, and the relationship between the score of the load and the load is expressed as a second monotonically increasing function; and
determining the score of each of the plurality of disks according to the score of the proportion, the weight of the proportion, the score of the load, and the weight of the load for each of the plurality of disks.
4. The method according to claim 2 , wherein the load of each of the plurality of disks is associated with a length of an input-output (IO) queue of each of the plurality of disks.
5. The method according to claim 1 , wherein determining the write-in disk according to the probabilities of the plurality of disks comprises:
determining a cumulative probability value of each of the plurality of disks according to the probabilities; and
performing a binary search on the cumulative probability values of the plurality of disks to determine a disk whose cumulative probability value satisfying a preset condition as the write-in disk,
wherein the preset condition includes disk i satisfying A i-1 <r≤A i , r being a random number, i being a natural number, and A i being a cumulative probability of disk i.
6. A data write-in apparatus, comprising:
a first selection unit configured to select at least one target replica server from a plurality of replica servers managed by a metadata server according to remaining capacity of the plurality of replica servers;
a second selection unit configured to select a write-in disk among a plurality of disks in the selected target replica server according to remaining capacity and load of each of the plurality of disks managed by the target replica server, wherein the second selection unit further comprises:
a probability determination subunit configured to determine a probability of a disk among the plurality of disks being selected according to the remaining capacity and the load of each of the plurality of disks managed by the target replica server; and
a disk determination subunit configured to determine the write-in disk according to the probabilities of the plurality of disks; and
a storage unit configured to store write-in data into the write-in disk through the target replica server.
7. The apparatus according to claim 6 , wherein the probability determination subunit further comprises:
a score determination subunit configured to determining a score of each of the plurality of disks according to a proportion of the remaining capacity to a total capacity of each of the plurality of disks, a weight of the remaining capacity of each of the plurality of disks, the load of each of the plurality of disks, and a weight of the load of each of the plurality of disks; and
a probability acquisition subunit configured to normalize the score of each of the plurality of disks to obtain the probability of each of the plurality of disks being selected.
8. The apparatus according to claim 7 , wherein the score determination subunit further comprises:
a first determination subunit configured to determine a score of the proportion of the remaining capacity to the total capacity and a score of the load of each of the plurality of disks, wherein the relationship between the score of the proportion and the proportion is expressed as a first monotonically increasing function and the relationship between the score of the load and the load is expressed as a second monotonically increasing function; and
a second determination subunit configured to determine the score of the disk according to the score of the proportion, the weight of the proportion, the score of the load, and the weight of the load for each of the plurality of disks.
9. The apparatus according to claim 6 , wherein the load of each of the plurality of disks is associated with a length of an input-output (IO) queue of each of the plurality of disks.
10. The apparatus according to claim 6 , wherein the disk determination subunit comprises:
an accumulation subunit configured to determine a cumulative probability value of each disk according to the probabilities; and
a search subunit configured to perform a binary search on the cumulative probability values of the plurality of disks to determine a disk whose cumulative probability value satisfying a preset condition as the write-in disk,
wherein the preset condition includes disk i satisfying A i-1 <r≤A i , r being a random number, i being a natural number, and A i being a cumulative probability of disk i.
11. A non-transitory computer readable medium storing a set of instructions that are executable by at least one processor of a computing system to cause the computing system to perform a data write-in method, the method comprising:
selecting at least one target replica server from a plurality of replica servers managed by a metadata server according to remaining capacity of the plurality of replica servers;
selecting a write-in disk among a plurality of disks in the selected target replica server according to remaining capacity and load of each of the plurality of disks managed by the target replica server, wherein selecting the write-in disk among the plurality of disks in the selected target replica server further comprises:
determining a probability of a disk among the plurality of disks being selected according to the remaining capacity and the load of each of the plurality of disks managed by the target replica server; and
determining the write-in disk according to the probabilities of the plurality of disks; and
storing write-in data into the write-in disk through the target replica server.
12. The non-transitory computer readable medium according to claim 11 , wherein determining the probability of the disk among the plurality of disks being selected comprises:
determining a score of each of the plurality of disks according to a proportion of the remaining capacity to a total capacity of each of the plurality of disks, a weight of the remaining capacity of each of the plurality of disks, the load of each of the plurality of disks, and a weight of the load of each of the plurality of disks; and
normalizing the score of each of the plurality of disks to obtain the probability of each of the plurality of disks being selected.
13. The non-transitory computer readable medium according to claim 12 , wherein determining the score of each of the plurality of disks comprises:
determining a score of the proportion of the remaining capacity to the total capacity of each of the plurality of disks, and a score of the load of each of the plurality of disks, wherein the relationship between the score of the proportion and the proportion is expressed as a first monotonically increasing function, and the relationship between the score of the load and the load is expressed as a first monotonically increasing function; and
determining the score of the disk according to the score of the proportion, the weight of the proportion, the score of the load, and the weight of the load for each of the plurality of disks.
14. The non-transitory computer readable medium according to claim 12 , wherein the load of each of the plurality of disks is associated with a length of an input-output (IO) queue of each of the plurality of disks.
15. The non-transitory computer readable medium according to claim 11 , wherein determining the write-in disk according to the probabilities of each disk includes:
determining a cumulative probability value of each of the plurality of disks according to the probabilities; and
performing a binary search on the cumulative probability values of the plurality of disks to determine a disk whose cumulative probability value satisfying a preset condition as the write-in disk,
wherein the preset condition includes disk i satisfying A i-1 <r≤A i , r being a random number, i being a natural number, and A i being a cumulative probability of disk i.
16. The method according to claim 3 , wherein the first monotonically increasing function and the second monotonically increasing function are the same function or different functions.
17. The method according to claim 1 , wherein
when the plurality of disks have the same load, one of the plurality of disks with a highest remaining capacity is selected as the write-in disk, and
when the plurality of disks have the same remaining capacity, one of the plurality of disks with lowest load is selected as the write-in disk.