Data processing method, computer device, and storage medium

1 . A data processing method, comprising:

detecting that a first storage file in an i-th layer of a target storage system meets a preset data merging condition, determining, in an (i+1)-th layer, a second storage position having a data merging relationship with a first storage position of the first storage file, and determining a second storage file located at the second storage position in the (i+1)-th layer, wherein i is an integer;

determining target write-in data to be written into a third storage file corresponding to the second storage file based on a data comparison result of first data in the first storage file and second data in the second storage file, and writing the target write-in data into the third storage file, wherein in response to that query keywords of the first data and query keywords of the second data are not completely same, the target write-in data comprises first target data and position information corresponding to second target data, wherein the first target data is data that only exists in the first storage file among the first storage file and the second storage file, and the second target data is data that only exists in the second storage file among the first storage file and the second storage file; and

determining a target storage virtual file corresponding to the second storage file, and establishing, in the target storage virtual file, index information corresponding to the third storage file, so as to establish an association relationship between the target storage virtual file and the third storage file; wherein index information stored in the target storage virtual file is used for querying data stored in storage files associated with the target storage virtual file, and the index information stored in the target storage virtual file comprises index information corresponding to the second storage file.

2 . The data processing method according to claim 1 ,

wherein the position information corresponding to the second target data is used for generating a first local index corresponding to the second target data.

3 . The data processing method according to claim 2 , further comprising:

determining a data combination corresponding to each same query keyword in the first data and the second data, wherein the data combination comprises update data in the first data and data to be updated in the second data; and

for the data combination corresponding to any one of the same query keywords, updating the data to be updated in the second data based on the update data in the data combination to obtain an updated second storage file.

4 . The data processing method according to claim 3 , wherein updating the data to be updated in the second data based on the update data in the data combination to obtain the updated second storage file comprises:

reorganizing the update data in the data combination and other data in the second data except the data to be updated, and writing reorganized data obtained after the reorganizing into the second storage file so as to update the data to be updated in the second storage file, to obtain the updated second storage file.

5 . The data processing method according to claim 3 , wherein before the index information corresponding to the third storage file is stored in the target storage virtual file, the data processing method further comprises:

determining a number of files associated with the target storage virtual file;

in response to that the number of the files is detected to be greater than a preset number, determining, among the index information stored in the target storage virtual file, target index information of which an adding time meets a preset adding time requirement; and

performing data merging on data in a target storage file corresponding to the target index information and other storage files except the target storage file, and deleting the target index information stored in the target storage virtual file.

6 . The data processing method according to claim 2 , further comprising:

determining, in the first data, update data which corresponds to a same query keyword as the second data, and writing the update data into a storage file associated with the target storage virtual file; and

establishing, in the target storage virtual file, a second local index corresponding to the update data so as to perform data query from the update data through the second local index.

7 . The data processing method according to claim 1 , wherein determining the target storage virtual file corresponding to the second storage file comprises:

taking a storage virtual file that stores index information corresponding to the second storage file, as the target storage virtual file, or in response to that there is no available storage virtual file for storing the index information corresponding to the second storage file, creating a target storage virtual file corresponding to the second storage file; and

the data processing method further comprises:

after creating the target storage virtual file corresponding to the second storage file, storing index information corresponding to the second storage file in the target storage virtual file, to establish an association relationship between the target storage virtual file and the second storage file.

8 . The data processing method according to claim 1 , wherein establishing, in the target storage virtual file, the index information corresponding to the third storage file comprises:

establishing, in the target storage virtual file, the index information corresponding to the third storage file, wherein the index information comprises a global index representing a position of the third storage file in the target storage system and a third local index representing a position of each data in the third storage file.

9 . The data processing method according to claim 8 , further comprising performing data query by:

determining, according to a received data query request, a query keyword corresponding to the received data query request;

determining, according to the query keyword, a target query virtual file including a global index corresponding to the query keyword from a plurality of storage virtual files and determining a storage file to be queried in the target query virtual file; and

acquiring query content matched with the query keyword from the storage file to be queried.

10 . The data processing method according to claim 1 , further comprising performing data query in the target storage system by:

determining, according to a received range query request, at least one storage virtual file corresponding to the received range query request;

in response to that the at least one storage virtual file is associated with a plurality of storage files, performing merge sort on iterators respectively corresponding to the plurality of storage files to obtain a target iterator corresponding to the at least one storage virtual file; and

traversing the storage files associated with the at least one storage virtual file based on the target iterator to determine a query result corresponding to the received range query request.

11 . A computer device, comprising: one or more processors, one or more memories, and a bus, wherein the memories store machine-readable instructions executable by the processors, the processors communicate with the memories via the bus when the computer device is running; and the machine-readable instructions, when executed by the processors, cause the processors to perform operations comprising:

in response to detecting that a first storage file in an i-th layer of a target storage system meets a preset data merging condition, determining, in an (i+1)-th layer, a second storage position having a data merging relationship with a first storage position of the first storage file, and determining a second storage file located at the second storage position in the (i+1)-th layer, wherein i is an integer;

determining target write-in data to be written into a third storage file corresponding to the second storage file based on a data comparison result of first data in the first storage file and second data in the second storage file, and writing the target write-in data into the third storage file, wherein in response to that query keywords of the first data and query keywords of the second data are not completely same, the target write-in data comprises first target data and position information corresponding to second target data, wherein the first target data is data that only exists in the first storage file among the first storage file and the second storage file, and the second target data is data that only exists in the second storage file among the first storage file and the second storage file; and

determining a target storage virtual file corresponding to the second storage file, and establishing, in the target storage virtual file, index information corresponding to the third storage file, so as to establish an association relationship between the target storage virtual file and the third storage file; wherein index information stored in the target storage virtual file is used for querying data stored in storage files associated with the target storage virtual file, and the index information stored in the target storage virtual file comprises index information corresponding to the second storage file.

12 . The computer device according to claim 11 , wherein the position information corresponding to the second target data is used for generating a first local index corresponding to the second target data.

13 . The computer device according to claim 12 , wherein the operations further comprise:

determining a data combination corresponding to each same query keyword in the first data and the second data, wherein the data combination comprises update data in the first data and data to be updated in the second data; and

for the data combination corresponding to any one of the same query keywords, updating the data to be updated in the second data based on the update data in the data combination to obtain an updated second storage file.

14 . The computer device according to claim 13 , wherein updating the data to be updated in the second data based on the update data in the data combination to obtain the updated second storage file comprises:

reorganizing the update data in the data combination and other data in the second data except the data to be updated, and writing the reorganized data obtained after the reorganizing into the second storage file so as to update the data to be updated in the second storage file, to obtain the updated second storage file.

15 . The computer device according to claim 13 , wherein before the index information corresponding to the third storage file is stored in the target storage virtual file, the operations further comprise:

determining a number of files associated with the target storage virtual file;

in response to that the number of the files is detected to be greater than a preset number, determining, among the index information stored in the target storage virtual file, target index information of which an adding time meets a preset adding time requirement; and

performing data merging on data in a target storage file corresponding to the target index information and other storage files except the target storage file, and deleting the target index information stored in the target storage virtual file.

16 . The computer device according to claim 12 , wherein the operations further comprise:

determining, in the first data, update data which corresponds to a same query keyword as the second data, and writing the update data into a storage file associated with the target storage virtual file; and

establishing, in the target storage virtual file, a second local index corresponding to the update data so as to perform data query from the update data through the second local index.

17 . The computer device according to claim 11 , wherein determining the target storage virtual file corresponding to the second storage file comprises:

taking a storage virtual file that stores index information corresponding to the second storage file, as the target storage virtual file, or in response to that there is no available storage virtual file for storing the index information corresponding to the second storage file, creating a target storage virtual file corresponding to the second storage file; and

the operations further comprise:

after creating the target storage virtual file corresponding to the second storage file, storing index information corresponding to the second storage file in the target storage virtual file, to establish an association relationship between the target storage virtual file and the second storage file.

18 . The computer device according to claim 11 , wherein establishing, in the target storage virtual file, the index information corresponding to the third storage file comprises:

establishing, in the target storage virtual file, the index information corresponding to the third storage file, wherein the index information comprises a global index representing a position of the third storage file in the target storage system and a third local index representing a position of each data in the third storage file.

19 . The computer device according to claim 18 , wherein the operations further comprise performing data query by:

determining, according to a received data query request, a query keyword corresponding to the received data query request;

determining, according to the query keyword, a target query virtual file including a global index corresponding to the query keyword from a plurality of storage virtual files and determining a storage file to be queried in the target query virtual file; and

acquiring query content matched with the query keyword from the storage file to be queried.

20 . A non-transitory computer readable storage medium, storing a computer program, wherein the computer program is executed by a processor to perform steps of the data processing method according to claim 1 .