Method and apparatus for processing data, reduction server, and mapping server
Methods and apparatuses for processing data are provided. In an implementation, a method comprising: obtaining, by a reduction server of a plurality of reduction servers in a distributed processing system and from a first storage area, metadata of first data to be read, wherein the first data comprises a target data block in a plurality of data blocks of second data, and wherein the second data comprises a processing result of a mapping server of a plurality of mapping servers for input data, determining a first address of the first data in a global memory based on the metadata, wherein the global memory comprises memories of the plurality of mapping servers and the plurality of reduction servers, and reading the first data from the global memory based on the first address.
1 . A method for processing data, comprising:
obtaining, by a reduction server of a plurality of reduction servers in a distributed processing system and from a first storage area, metadata of first data to be read, wherein the first data comprises a target data block in a plurality of data blocks of second data, and wherein the second data comprises a processing result of a mapping server of a plurality of mapping servers for input data, and the plurality of data blocks of the second data are generated based on operations comprising:
partitioning, by the mapping server in response to determining that data in the plurality of data blocks needs to be sorted, a second storage area in a global memory into a plurality of sub-areas based on a preset second size; and
storing, by the mapping server, the plurality of data blocks into the plurality of sub-areas according to a sequence of the plurality of sub-areas;
determining, by the reduction server, a first address of the first data in the global memory based on the metadata, wherein the global memory comprises memories of the plurality of mapping servers and the plurality of reduction servers; and
reading, by the reduction server, the first data from the global memory based on the first address.
2 . The method according to claim 1 , wherein the reading the first data from the global memory based on the first address comprises:
mapping, by the reduction server, the first address to a second address, wherein the first address is outside an access range of the reduction server and the second address is within the access range of the reduction server; and
reading, by the reduction server, the first data from the global memory based on the second address.
3 . The method according to claim 1 , wherein the method further comprises:
performing, by the reduction server, registration to add a memory of the reduction server to the global memory, by using a preset register instruction after the reduction server is connected to the distributed processing system.
4 . A method for processing data, comprising:
processing, by a mapping server of a plurality of mapping servers in a distributed processing system, input data to obtain second data;
partitioning, by the mapping server, the second data into a plurality of data blocks based on a preset label; and
storing, by the mapping server, the plurality of data blocks into a second storage area in a global memory, comprising:
partitioning, by the mapping server in response to determining that data in the plurality of data blocks needs to be sorted, the second storage area into a plurality of sub-areas based on a preset second size; and
storing, by the mapping server, the plurality of data blocks into the plurality of sub-areas according to a sequence of the plurality of sub-areas,
wherein the global memory comprises memories of the plurality of mapping servers and a plurality of reduction servers.
5 . The method according to claim 4 , wherein the storing the plurality of data blocks into a second storage area further comprises:
sorting, by the mapping server in response to determining that the plurality of data blocks are stored into the plurality of sub-areas in sequence, data in the sub-areas in which storage has been performed by updating an ordered index linked list, wherein the ordered index linked list is sorted by linking a location index of data by using a linked list.
6 . The method according to claim 4 , wherein the mapping server comprises at least one first operator that processes the input data, and wherein the method further comprises:
applying, in an initialization phase of a first operation process performed on the mapping server, to the global memory for the second storage area based on a quantity of processor cores of the mapping server, so that each processor core corresponds to one second storage area, wherein at least one first operator runs on each processor core.
7 . The method according to claim 4 , wherein the partitioning the second data into a plurality of data blocks based on a preset label comprises:
partitioning, by the mapping server based on a hash, the second data into a plurality of data blocks based on the preset label.
8 . The method according to claim 4 , wherein the storing the plurality of data blocks into a second storage area comprises:
determining, by the mapping server, a third address of the second storage area;
mapping, by the mapping server, the third address to a fourth address, wherein the third address is outside an access range of the mapping server and the fourth address is within the access range of the mapping server; and
storing, by the mapping server, the plurality of data blocks into the second storage area based on the fourth address.
9 . The method according to claim 4 , wherein the method further comprises:
determining, by the mapping server, metadata of the plurality of data blocks; and
storing, by the mapping server, the metadata of the plurality of data blocks into a first storage area.
10 . The method according to claim 4 , wherein the method further comprises:
performing, by the mapping server, registration by using a preset register instruction to add a memory of the mapping server to the global memory, after the mapping server is connected to the distributed processing system.
11 . A distributed processing system, comprising a plurality of mapping servers and a plurality of reduction servers, and memories of the plurality of mapping servers and memories of the plurality of reduction servers form a global memory; wherein
a reduction server of the plurality of reduction servers is configured to:
obtain, from a first storage area, metadata of first data to be read, wherein the first data comprises a target data block in a plurality of data blocks of second data, and wherein the second data comprises a processing result of a mapping server of a plurality of mapping servers for input data;
determine a first address of the first data in the global memory based on the metadata; and
read the first data from the global memory based on the first address; and
a mapping server of the plurality of mapping servers is configured to:
process input data to obtain the second data;
partition the second data into a plurality of data blocks based on a preset label; and
store the plurality of data blocks into a second storage area in the global memory, wherein:
partition, in response to determining that data in the plurality of data blocks needs to be sorted, the second storage area into a plurality of sub-areas based on a preset second size; and
store the plurality of data blocks into the plurality of sub-areas according to a sequence of the plurality of sub-areas.
12 . The distributed processing system according to claim 11 , wherein the reduction server is configured to:
map the first address to a second address, wherein the first address is outside an access range of the reduction server and the second address is within the access range of the reduction server; and
read the first data from the global memory based on the second address.
13 . The distributed processing system according to claim 11 , wherein the reduction server is further configured to:
perform registration to add a memory of the reduction server to the global memory, by using a preset register instruction after the reduction server is connected to the distributed processing system.
14 . The distributed processing system according to claim 13 , wherein the mapping server is further configured to:
sort, in response to determining that the plurality of data blocks are stored into the plurality of sub-areas in sequence, data in the sub-areas in which storage has been performed by updating an ordered index linked list, wherein the ordered index linked list is sorted by linking a location index of data by using a linked list.
15 . The distributed processing system according to claim 11 , wherein the mapping server is further configured to:
apply, in an initialization phase of a first operation process performed on the mapping server, to the global memory for the second storage area based on a quantity of processor cores of the mapping server, so that each processor core corresponds to one second storage area, wherein at least one first operator runs on each processor core and the first operation process runs on the mapping server to process the input data.
16 . The distributed processing system according to claim 14 , wherein mapping server is configured to:
partition, based on a hash manner, the second data into a plurality of data blocks based on the preset label.
17 . The distributed processing system according to claim 14 , wherein the mapping server is configured to:
determine a third address of the second storage area;
map the third address to a fourth address, wherein the third address is outside an access range of the mapping server and the fourth address is within the access range of the mapping server; and
store the plurality of data blocks into the second storage area based on the fourth address.
18 . The distributed processing system according to claim 14 , wherein the mapping server is further configured to:
determine metadata of the plurality of data blocks; and
store the metadata of the plurality of data blocks into the first storage area.
19 . The distributed processing system according to claim 14 , wherein the mapping server is further configured to:
perform registration by using a preset register instruction to add a memory of the mapping server to the global memory, after the mapping server is connected to the distributed processing system.
20 . The distributed processing system according to claim 14 , wherein the mapping server is further configured to:
determine, in response to a first memory satisfies a first condition, first target data from data stored in the first memory; and
store the first target data into an external storage area, wherein the first condition is that used space of the first memory is greater than or equal to a first threshold, or a ratio of the used space of the first memory to total space of the first memory is greater than or equal to a second threshold, and wherein the first memory is the global memory or a part of the global memory.