Partition splitting method and apparatus, electronic device, and readable storage medium

1. A computer-implemented method, comprising:

dividing data files in a target source partition into at least two data groups;

selecting a first data group from the at least two data groups as a target splitting data group;

performing a first splitting step on the target splitting data group, wherein the first splitting step comprises:

splitting the target splitting data group into a plurality of data sub-groups based on a predetermined splitting rule, wherein each data sub-group of the plurality of data sub-groups corresponds to a first new partition of at least two new partitions for the target source partition, and

assigning each data sub-group to the first new partition corresponding to the data sub-group;

selecting, as the target splitting data group, a second data group from the at least two data groups, the second data group being different from the first data group; and

performing a merging processing on target data in the first new partition, wherein the target data comprises a plurality of data blocks, each data block comprises a plurality of data entries, and wherein performing the merging processing on the target data in the first new partition comprises:

creating a target file in the first new partition;

determining whether there is an intersection among data blocks in the target data;

in response to determining that there is no intersection between the data blocks in the target data, storing the data blocks in the target file; and

in response to determining that there is an intersection among the data blocks in the target data, successively iterating the data entries in each data block in the target data, and in an iterative process, fusing data entries having an intersection into a new data entry and storing the data entry in the target file.

2. The computer-implemented method of claim 1 , further comprising:

after assigning each data sub-group to the first new partition corresponding to the data sub-group, deleting the target splitting data group that is split from the target source partition.

3. The computer-implemented method of claim 1 , further comprising:

before dividing the data files in the target source partition into the at least two data groups, creating a reference file in each new partition of the at least two new partitions for one of the data files stored in the target source partition, wherein the reference file is used to reference the one of the data files; and

after assigning each data sub-group to the first new partition corresponding to the data sub-group, deleting the reference file from the first new partition after data corresponding to the reference file is all split.

4. The computer-implemented method of claim 1 , wherein the data files are stored by a database system using a log-structured merge (LSM) tree and a first data file of the data files is stored in the target source partition, and the first data file comprising first data with a volume that exceeds a predetermined value.

5. The computer-implemented method of claim 4 , wherein a second data file is further stored in the target source partition, the second data file comprises second data with a volume that does not exceed the predetermined value, and the method further comprises:

performing a second splitting step on the second data file to split the second data file into the at least two new partitions, wherein the second splitting step comprises:

splitting the second data file into a plurality of third data groups based on the predetermined splitting rule, wherein each third data group corresponds to a second new partition among the at least two new partitions; and

assigning each third data group to the second new partition corresponding to the third data group.

6. The computer-implemented method of claim 4 , wherein the first data file is a baseline data file, and the target data comprises baseline data and dump data in the first new partition, and the baseline data comprises data obtained by splitting the current target splitting data group and data obtained by splitting the previous target splitting data group and through merging, and the dump data comprises data obtained by splitting a dump data file in the target source partition and/or data obtained by performing freezing and dump processing on new data written into the first new partition.

7. The computer-implemented method of claim 1 , comprising:

performing the merging processing on the target data in the first new partition, wherein the target data comprises baseline data and dump data in the first new partition, the baseline data and the dump data each comprises the plurality of data blocks, each data block comprises the plurality of data entries, and wherein performing the merging processing on the target data in the first new partition comprises:

in response to determining that there is no intersection between the data blocks comprised in the target data, directly putting all the data blocks comprised in the baseline data into the target file, and successively iterating the data entries in the dump data into the target file; and

in response to determining that there is an intersection between the data blocks comprised in the target data, successively iterating the data entries in each data block in each of the baseline data and the dump data into the target file, and in the iteration process, fusing the data entries having an intersection into the new data entry and then putting the data entry into the target file.

8. The computer-implemented method of claim 4 , further comprising:

after assigning each second data group to the first new partition corresponding to the second data group, deleting the target splitting data group that is split from the target source partition.

9. The computer-implemented method of claim 4 , further comprising:

before performing the first splitting step on the first data file, creating a reference file in each new partition for each data file stored in the target source partition, wherein each reference file is used to reference a corresponding data file; and

after assigning each second data group to the first new partition corresponding to the second data group, deleting the reference file from the first new partition after the data file corresponding to the reference file is split.

10. The computer-implemented method of claim 9 , further comprising:

before performing the first splitting step on the first data file to split the first data file into the at least two new partitions, determining whether the target source partition satisfies a predetermined dump condition;

in response to determining that the target source partition satisfies the predetermined dump condition, performing dump processing on data in the target source partition; and

after the target source partition is dumped, updating the reference file in each new partition corresponding to the target source partition.

11. A non-transitory, computer-readable medium storing one or more instructions executable by a computer system to perform operations comprising:

dividing data files in a target source partition into at least two data groups;

selecting a first data group from the at least two data groups as a target splitting data group;

performing a first splitting step on the target splitting data group, wherein the first splitting step comprises:

splitting the target splitting data group into a plurality of data sub-groups based on a predetermined splitting rule, wherein each data sub-group of the plurality of data sub-groups corresponds to a first new partition of at least two new partitions for the target source partition, and

assigning each data sub-group to the first new partition corresponding to the data sub-group;

selecting, as the target splitting data group, a second data group from the at least two data groups, the second data group being different from the first data group; and

performing a merging processing on target data in the first new partition, wherein the target data comprises a plurality of data blocks, each data block comprises a plurality of data entries, and wherein performing the merging processing on the target data in the first new partition comprises:

creating a target file in the first new partition;

determining whether there is an intersection among data blocks in the target data;

in response to determining that there is no intersection between the data blocks in the target data, storing the data blocks in the target file; and

in response to determining that there is an intersection among the data blocks in the target data, successively iterating the data entries in each data block in the target data, and in an iterative process, fusing data entries having an intersection into a new data entry and storing the data entry in the target file.

12. The non-transitory, computer-readable medium of claim 11 , wherein the operations further comprise:

after assigning each data sub-group to the first new partition corresponding to the data sub-group, deleting the target splitting data group that is split from the target source partition.

13. The non-transitory, computer-readable medium of claim 11 , wherein the operations further comprise:

before dividing the data files in the target source partition into the at least two data groups, creating a reference file in each new partition of the at least two new partitions for one of the data files stored in the target source partition, wherein the reference file is used to reference the one of the data files; and

after assigning each data sub-group to the first new partition corresponding to the data sub-group, deleting the reference file from the first new partition after data corresponding to the reference file is all split.

14. The non-transitory, computer-readable medium of claim 11 , wherein the data files are stored by a database system using a log-structured merge (LSM) tree and a first data file of the data files is stored in the target source partition, and the first data file comprising first data with a volume that exceeds a predetermined value.

15. The non-transitory, computer-readable medium of claim 14 , wherein a second data file is further stored in the target source partition, the second data file comprises second data with a volume that does not exceed the predetermined value, and wherein the operations further comprise:

performing a second splitting step on the second data file to split the second data file into the at least two new partitions, wherein the second splitting step comprises:

splitting the second data file into a plurality of third data groups based on the predetermined splitting rule, wherein each third data group corresponds to a second new partition among the at least two new partitions; and

assigning each third data group to the second new partition corresponding to the third data group.

16. The non-transitory, computer-readable medium of claim 14 , wherein the first data file is a baseline data file, and the target data comprises baseline data and dump data in the first new partition, and the baseline data comprises data obtained by splitting the current target splitting data group and data obtained by splitting the previous target splitting data group and through merging, and the dump data comprises data obtained by splitting a dump data file in the target source partition and/or data obtained by performing freezing and dump processing on new data written into the first new partition.

17. The non-transitory, computer-readable medium of claim 11 , the operations comprise:

performing the merging processing on the target data in the first new partition, wherein the target data comprises baseline data and dump data in the first new partition, the baseline data and the dump data each comprises the plurality of data blocks, each data block comprises the plurality of data entries, and wherein performing the merging processing on the target data in the first new partition comprises:

in response to determining that there is no intersection between the data blocks comprised in the target data, directly putting all the data blocks comprised in the baseline data into the target file, and successively iterating the data entries in the dump data into the target file; and

in response to determining that there is an intersection between the data blocks comprised in the target data, successively iterating the data entries in each data block in each of the baseline data and the dump data into the target file, and in the iteration process, fusing the data entries having an intersection into the new data entry and then putting the data entry into the target file.

18. The non-transitory, computer-readable medium of claim 14 , wherein the operations further comprise:

after assigning each second data group to the first new partition corresponding to the second data group, deleting the target splitting data group that is split from the target source partition.

19. The non-transitory, computer-readable medium of claim 14 , wherein the operations further comprise:

before performing the first splitting step on the first data file, creating a reference file in each new partition for each data file stored in the target source partition, wherein each reference file is used to reference a corresponding data file; and

after assigning each second data group to the first new partition corresponding to the second data group, deleting the reference file from the first new partition after the data file corresponding to the reference file is split.

20. The non-transitory, computer-readable medium of claim 19 , wherein the operations further comprise:

before performing the first splitting step on the first data file to split the first data file into the at least two new partitions, determining whether the target source partition satisfies a predetermined dump condition;

in response to determining that the target source partition satisfies the predetermined dump condition, performing dump processing on data in the target source partition; and

after the target source partition is dumped, updating the reference file in each new partition corresponding to the target source partition.

21. A computer-implemented system, comprising:

one or more computers; and

one or more computer memory devices interoperably coupled with the one or more computers and having tangible, non-transitory, machine-readable media storing one or more instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

dividing data files in a target source partition into at least two data groups;

selecting a first data group from the at least two data groups as a target splitting data group;

performing a first splitting step on the target splitting data group, wherein the first splitting step comprises:

splitting the target splitting data group into a plurality of data sub-groups based on a predetermined splitting rule, wherein each data sub-group of the plurality of data sub-groups corresponds to a first new partition of at least two new partitions for the target source partition, and

assigning each data sub-group to the first new partition corresponding to the data sub-group;

selecting, as the target splitting data group, a second data group from the at least two data groups, the second data group being different from the first data group; and

performing a merging processing on target data in the first new partition, wherein the target data comprises a plurality of data blocks, each data block comprises a plurality of data entries, and wherein performing the merging processing on the target data in the first new partition comprises:

creating a target file in the first new partition;

determining whether there is an intersection among data blocks in the target data;

in response to determining that there is no intersection between the data blocks in the target data, storing the data blocks in the target file; and

in response to determining that there is an intersection among the data blocks in the target data, successively iterating the data entries in each data block in the target data, and in an iterative process, fusing data entries having an intersection into a new data entry and storing the data entry in the target file.

22. The computer-implemented system of claim 21 , wherein the operations further comprise:

after assigning each data sub-group to the first new partition corresponding to the data sub-group, deleting the target splitting data group that is split from the target source partition.

23. The computer-implemented system of claim 21 , wherein the operations further comprise:

before dividing the data files in the target source partition into the at least two data groups, creating a reference file in each new partition of the at least two new partitions for one of the data files stored in the target source partition, wherein the reference file is used to reference the one of the data files; and

after assigning each data sub-group to the first new partition corresponding to the data sub-group, deleting the reference file from the first new partition after data corresponding to the reference file is all split.

24. The computer-implemented system of claim 21 , wherein the data files are stored by a database system using a log-structured merge (LSM) tree and a first data file of the data files is stored in the target source partition, and the first data file comprising first data with a volume that exceeds a predetermined value.

25. The computer-implemented system of claim 24 , wherein a second data file is further stored in the target source partition, the second data file comprises second data with a volume that does not exceed the predetermined value, and wherein the operations further comprise:

performing a second splitting step on the second data file to split the second data file into the at least two new partitions, wherein the second splitting step comprises:

splitting the second data file into a plurality of third data groups based on the predetermined splitting rule, wherein each third data group corresponds to a second new partition among the at least two new partitions; and

assigning each third data group to the second new partition corresponding to the third data group.

26. The computer-implemented system of claim 24 , wherein the first data file is a baseline data file, and the target data comprises baseline data and dump data in the first new partition, and the baseline data comprises data obtained by splitting the current target splitting data group and data obtained by splitting the previous target splitting data group and through merging, and the dump data comprises data obtained by splitting a dump data file in the target source partition and/or data obtained by performing freezing and dump processing on new data written into the first new partition.

27. The computer-implemented system of claim 21 , the operations comprise:

performing the merging processing on the target data in the first new partition, wherein the target data comprises baseline data and dump data in the first new partition, the baseline data and the dump data each comprises the plurality of data blocks, each data block comprises the plurality of data entries, and wherein performing the merging processing on the target data in the first new partition comprises:

in response to determining that there is no intersection between the data blocks comprised in the target data, directly putting all the data blocks comprised in the baseline data into the target file, and successively iterating the data entries in the dump data into the target file; and

in response to determining that there is an intersection between the data blocks comprised in the target data, successively iterating the data entries in each data block in each of the baseline data and the dump data into the target file, and in the iteration process, fusing the data entries having an intersection into the new data entry and then putting the data entry into the target file.

28. The computer-implemented system of claim 24 , wherein the operations further comprise:

after assigning each second data group to the first new partition corresponding to the second data group, deleting the target splitting data group that is split from the target source partition.

29. The computer-implemented system of claim 24 , wherein the operations further comprise:

before performing the first splitting step on the first data file, creating a reference file in each new partition for each data file stored in the target source partition, wherein each reference file is used to reference a corresponding data file; and

after assigning each second data group to the first new partition corresponding to the second data group, deleting the reference file from the first new partition after the data file corresponding to the reference file is split.

30. The computer-implemented system of claim 29 , wherein the operations further comprise:

before performing the first splitting step on the first data file to split the first data file into the at least two new partitions, determining whether the target source partition satisfies a predetermined dump condition;

in response to determining that the target source partition satisfies the predetermined dump condition, performing dump processing on data in the target source partition; and

after the target source partition is dumped, updating the reference file in the each new partition corresponding to the target source partition.