Data I/O processing method and apparatus, storage medium, and device
View Patent ↗A data I/O processing method includes obtaining data requested consisting of a plurality of basic data blocks in sequence; grouping the data in sequence to obtain a plurality of segmented data blocks in sequence; sequentially determining whether each segmented data block has a time delay statistical record based on the time for completion of the operation processing of a previous basic data block; in response to the time delay statistical record being present in the segmented data block, setting a waiting time period according to the time delay statistical record; sequentially merging the basic data blocks, which have not been subjected to the operation processing, in the segmented data blocks within the waiting time period until the waiting time period is ended or the merged basic data blocks reach the size of the segmentation unit, and stopping merging; and sending the merged basic data blocks and performing the operation processing.
1. A data I/O processing method, applied by a distributed storage system, comprising the following steps:
obtaining data which is requested to be subjected to operation processing and consists of a plurality of basic data blocks in sequence;
grouping the data in sequence by taking a preset segmentation unit as a unit to obtain a plurality of segmented data blocks in sequence;
sequentially processing and determining whether each segmented data block has a time delay statistical record that is based on the time for completion of the operation processing of a previous basic data block, wherein the time for the completion of the operation processing of the previous basic data block is a time period that a basic data block in a first area of a disk is sent to a second area of the disk for an operation, including a read operation or a write operation or a deleting operation, and then returns back to the first area of the disk after the operation is ended;
in response to the time delay statistical record being present in the segmented data block, setting a waiting time period according to the time delay statistical record;
sequentially merging the basic data blocks, which have not been subjected to the operation processing, in the segmented data blocks within the waiting time period until the waiting time period is ended or the waiting time period is not ended but the merged basic data blocks reach the size of the segmentation unit, and stopping merging; and
sending the merged basic data blocks to a storage area corresponding to the segmented data block thereof, and performing the operation processing, so as to shorten the time for data input and output (I/O), such that the throughput of the distributed storage system is increased;
wherein each storage area is established in advance, the storage capacity of each storage area is the same as the size of the segmentation unit, and a mapping relationship between each segmented data block and the corresponding storage area is established.
2. The method according to claim 1 , wherein the step of sequentially processing and determining whether each segmented data block has a time delay statistical record that is based on the time for completion of the operation processing of a previous basic data block comprises:
determining whether the first segmented data block has a time delay statistical record that is based on the time for completion of the operation processing of previous several basic data blocks; and
determining whether the nth segmented data block has a time delay statistical record that is based on the time for completion of the operation processing of basic data blocks in the nth segmented data block and/or in previous n−1 segmented data blocks, wherein n is an integer greater than 1.
3. The method according to claim 1 , wherein the time delay statistical record is the average time for completion of the operation processing of a plurality of previous basic data blocks.
4. The method according to claim 1 , wherein the step of sequentially merging the basic data blocks, which have not been subjected to the operation processing, in the segmented data blocks within the waiting time period until the waiting time period is ended or the waiting time period is not ended but the merged basic data blocks reach the size of the segmentation unit, and stopping merging comprises:
in a case that the segmented data block is the first segmented data block, sequentially merging the basic data blocks, which have not been subjected to the operation processing, in the first segmented data block within the waiting time period until the waiting time period is ended, and stopping merging; and
in a case that the segmented data block is the nth segmented data block, sequentially merging the basic data blocks, which have not been subjected to the operation processing, in the nth segmented data block within the waiting time period until the waiting time period is ended or the waiting time period is not ended but the merged basic data blocks reach the size of the segmentation unit, and stopping merging, wherein n is an integer greater than 1.
5. The method according to claim 1 , further comprising:
in response to the time delay statistical record being not present in the segmented data block, directly sending all of the basic data blocks in the segmented data block, and performing the operation processing.
6. The method according to claim 1 , further comprising:
after obtaining the data which is requested to be subjected to the operation processing, determining whether there is other data, which has the same offset and length as the obtained data, being subjected to the operation processing;
in response to said other data being subjected to the operation processing, putting the obtained data in a blocking queue; and
in response to no other data being subjected to the operation processing, advancing to the step of grouping the data in sequence by taking a preset segmentation unit as a unit.
7. A non-transitory computer-readable storage medium, having computer program instructions stored thereon, wherein, when the computer program instructions are executed, the method according to claim 1 is implemented.
8. A computer device, comprising a memory and a processor, wherein the memory has a computer program stored thereon, and when the computer program is executed by the processor, the method according to claim 1 is implemented.
9. The method according to claim 3 , wherein said average time is the average of the sum of the time for completion of the operation processing of each basic data block.
10. The method according to claim 6 , wherein the complexity of the merging processing is reduced by blocking request data with the same offset and length.
11. The method according to claim 1 , wherein a storage system only performs one operation at one moment at a same position, and the system performs resending only when the operation times out or when there is an error.
12. The method according to claim 1 , wherein, when segmented data blocks have no time statistical record, the basic data blocks thereof are sent in sequence without merging processing.
13. The method according to claim 1 , wherein the segmented data blocks are arranged in sequence according to the sequence of basic data blocks in the data which is requested to be subjected to the operation processing.
14. The method according to claim 13 , wherein the first one or several basic data blocks form a first time delay statistical record, and the operation processing of the first one or several basic data blocks is firstly completed.
15. The method according to claim 4 , wherein the first segmented data block has basic data blocks of which the operation processing is completed, such that a situation that the merged basic data blocks reach the size of the segmentation unit does not occur in the first segmented data block, and a situation that the merged basic data blocks reach the size of the segmentation unit occurs in subsequent segmented data blocks.
16. The method according to claim 2 , wherein the time delay statistical record of the nth segmented data block is obtained based on the basic data blocks of which the operation processing is completed in the nth segmented data block, and/or, in previous segmented data blocks.