Mapping and reducing
View Patent ↗As disclosed herein, a system for conducting mapping and reducing operations includes a shared storage subsystem that is connected to one or more mapping servers and one or more reducing servers via a high-speed data link and communication protocol. Each mapping server receives a multitude of data records, aggregates the data records having a particular value, and sorts and stores the resulting aggregated data records on the shared storage subsystem. Each reducing server accesses the shared storage subsystem and accumulates information on the aggregated data records for a particular common value. In many instances, the access rates to the shared storage subsystem achieved by the mapping servers and the reducing servers approach that of accessing a local attached storage device. A computer program product and method corresponding to the system for conducting mapping and reducing operations are also disclosed herein.
1. A method for optimizing data communications between servers, the method comprising:
receiving, by a task tracker, a control information comprising a storage pointer of a location of aggregated data records via a first communication pathway from a first server, wherein the aggregated data records are stored on a shared storage subsystem by the first server via a second communication pathway;
transmitting, by the task tracker, the storage pointer to a second server via the first communication pathway; and
transmitting, by the task tracker, a notification to the second server indicating that the aggregated data is ready for use and for the second server to retrieve the data from the shared storage subsystem via the second communication pathway.
2. The method of claim 1 , wherein the first communication pathway utilizes TCP/IP (Transport Control Protocol/Internet Protocol).
3. The method of claim 1 , wherein the aggregated data records have a common value for a selected field within the data records to provide aggregated data records for each common value.
4. The method of claim 1 , wherein the storage pointer further comprises a filename and an offset reference.
5. The method of claim 1 , wherein the notification further comprises a control message indicating when the mapping operations have been completed.
6. The method of claim 1 , wherein the second communication pathway utilizes iSER (iSCSI Extensions for RDMA) network is a TCP/IP network.
7. The system of claim 1 , wherein the first server is a mapping server and the second server is a reducing server.
8. A computer program product for conducting mapping and reducing operations, the computer program product comprising:
one or more computer readable storage media and program instructions stored on the one or more computer readable storage media, the program instructions comprising instructions to:
program instructions to receive a control information comprising a storage pointer of a location of aggregated data records via a first communication pathway from a first server, wherein the aggregated data records are stored on a shared storage subsystem by the first server via a second communication pathway;
transmitting, by the task tracker, the storage pointer to a second server via the first communication pathway; and
transmitting, by the task tracker, a notification to the second server indicating that the aggregated data is ready for use and for the second server to retrieve the data from the shared storage subsystem via the second communication pathway.
9. The computer program product of claim 8 , wherein the instructions further comprise accessing the shared storage subsystem with a storage access protocol the first communication pathway utilizes TCP/IP (Transport Control Protocol/Internet Protocol).
10. The computer program product of claim 8 , wherein the aggregated data records have a common value for a selected field within the data records to provide aggregated data records for each common value.
11. The computer program product of claim 8 , wherein the storage pointer further comprises a filename and an offset reference.
12. The computer program product of claim 8 , wherein the notification further comprises a control message indicating when the mapping operations have been completed.
13. The computer program product of claim 8 , wherein the second communication pathway utilizes iSER (iSCSI Extensions for RDMA).