Data processing method, apparatus, electronic device, and computer storage medium
A data processing method including: acquiring information of multiple first data blocks of a first level to be merged and information of multiple second data blocks of a second level to be merged, wherein each of the first data blocks of the first level has the same first size, and each of the second data blocks of the second level has the same second size; performing, according to the information of the multiple first data blocks and the information of the multiple second data blocks, data comparison on each of the first data blocks and each of the second data blocks; and performing, according to a comparison result, data compaction on the first level and the second level. The embodiments of the present disclosure effectively reduce system overheads caused by I/O data read and write and data comparison, thereby improving the system throughput.
1. A method comprising:
acquiring information of multiple first data blocks of a first level to be merged and information of multiple second data blocks of a second level to be merged, each of the multiple first data blocks of the first level having a same first size, each of the multiple second data blocks of the second level having a same second size, and the first size being different from the second size;
performing, according to the information of the multiple first data blocks and the information of the multiple second data blocks, data comparison on each of the multiple first data blocks and each of the multiple second data blocks;
performing, according to a comparison result, data compaction on the first level and the second level, wherein the performing, according to the comparison result, data compaction on the first level and the second level comprises:
determining, according to the comparison result, that a data overlapping exists between the multiple first data blocks and the multiple second data blocks; and
merging a first data block having overlapped data with an overlapping second data block, wherein the merging the first data block having overlapped data with the overlapping second data block comprises:
determining that a data gap meeting a preset criteria exists in data of the first data block having the overlapped data; and
dividing, according to the data gap, the first data block into at least two data blocks, and performing merging with the overlapping second data block; and
storing compacted data of each level in fixed-sized data blocks on a magnetic disk.
2. The method according to claim 1 , wherein: the second level is a higher level adjacent to the first level.
3. The method according to claim 2 , wherein the second size is smaller than the first size.
4. The method according to claim 1 , wherein the performing, according to the comparison result, data compaction on the first level and the second level comprises:
determining, according to the comparison result, that a data overlapping does not exist between the multiple first data blocks and the multiple second data blocks; and
reusing, in the second level, a first data block without overlapping.
5. The method according to claim 1 , wherein both the first size and the second size are sizes meeting a preset data overlapping probability.
6. The method according to claim 5 , wherein both the first size and the second size have a size range of 2M-16M.
7. The method according to claim 1 , wherein:
both the multiple first data blocks and the multiple second data blocks are divided into data pages of a preset size according to a preset rule.
8. The method according to claim 7 , wherein:
the performing, according to the information of the multiple first data blocks and the information of the multiple second data blocks, data comparison on each of the multiple first data blocks and each of the multiple second data blocks comprises:
reading, according to the information of the multiple first data blocks and the information of the multiple second data blocks, data of each of the multiple first data blocks and data of each of the multiple second data blocks by using a data page as a unit; and
performing, according to a reading result, data comparison on each of the multiple first data blocks and each of the multiple second data blocks.
9. The method according to claim 1 , wherein the acquiring the information of the multiple first data blocks of the first level to be merged and the information of the multiple second data blocks of the second level to be merged comprises:
acquiring, according to level indexes, the information of the multiple first data blocks of the first level to be merged and the information of the multiple second data blocks of the second level to be merged.
10. The method according to claim 9 , further comprising:
generating the level indexes according to information of each data block of each level and information of each data file formed by data blocks.
11. An apparatus comprising:
one or more processors; and
one or more memories storing thereon computer-readable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising:
acquiring information of multiple first data blocks of a first level to be merged and information of multiple second data blocks of a second level to be merged, each of the multiple first data blocks of the first level having a same first size, each of the multiple second data blocks of the second level having a same second size, and the first size being different from the second size;
performing, according to the information of the multiple first data blocks and the information of the multiple second data blocks, data comparison on each of the multiple first data blocks and each of the multiple second data blocks;
performing, according to a comparison result, data compaction on the first level and the second level, wherein the performing, according to the comparison result, data compaction on the first level and the second level comprises:
determining, according to the comparison result, that a data overlapping exists between the multiple first data blocks and the multiple second data blocks; and
merging a first data block having overlapped data with an overlapping second data block, wherein the merging the first data block having overlapped data with the overlapping second data block comprises:
determining that a data gap meeting a preset criteria exists in data of the first data block having the overlapped data; and
dividing, according to the data gap, the first data block into at least two data blocks, and performing merging with the overlapping second data block; and
storing compacted data of each level in fixed-sized data blocks on a magnetic disk.
12. The apparatus according to claim 11 , wherein:
the second level is a higher level adjacent to the first level; and
the second size is smaller than the first size.
13. The apparatus according to claim 11 , wherein:
both the multiple first data blocks and the multiple second data blocks are divided into data pages of a preset size according to a preset rule, and
the performing, according to the information of the multiple first data blocks and the information of the multiple second data blocks, data comparison on each of the multiple first data blocks and each of the multiple second data blocks comprises:
reading, according to the information of the multiple first data blocks and the information of the multiple second data blocks, data of each of the multiple first data blocks and data of each of the multiple second data blocks by using a data page as a unit; and
performing, according to a reading result, data comparison on each of the multiple first data blocks and each of the multiple second data blocks.
14. The apparatus according to claim 11 , wherein the acquiring the information of the multiple first data blocks of the first level to be merged and the information of the multiple second data blocks of the second level to be merged comprises:
acquiring, according to level indexes, the information of the multiple first data blocks of the first level to be merged and the information of the multiple second data blocks of the second level to be merged.
15. One or more memories storing thereon computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising:
acquiring information of multiple first data blocks of a first level to be merged and information of multiple second data blocks of a second level to be merged, each of the first data blocks of the first level having a same first size, each of the second data blocks of the second level having a same second size, and the first size being different from the second size;
performing, according to the information of the multiple first data blocks and the information of the multiple second data blocks, data comparison on each of the multiple first data blocks and each of the multiple second data blocks;
performing, according to a comparison result, data compaction on the first level and the second level, wherein the performing, according to the comparison result, data compaction on the first level and the second level comprises:
determining, according to the comparison result, that a data overlapping exists between the multiple first data blocks and the multiple second data blocks; and
merging a first data block having overlapped data with an overlapping second data block, wherein the merging the first data block having overlapped data with the overlapping second data block comprises:
determining that a data gap meeting a preset criteria exists in data of the first data block having the overlapped data; and
dividing, according to the data gap, the first data block into at least two data blocks, and performing merging with the overlapping second data block; and
storing compacted data of each level in fixed-sized data blocks on a magnetic disk.