Memory optimized data shuffle

1. A method comprising:

maintaining a first data shuffle memory pool at a data shuffle writer node and a second data shuffle memory pool at a data shuffle reader node, wherein the data shuffle writer node and the data shuffle reader node are part of a set of multiple nodes of a distributed data processing system;

performing an in-memory compression on at least a portion of a data set from the first data shuffle memory pool;

performing a data shuffle operation on the at least a portion of the compressed data from the first shuffle memory pool, wherein the data shuffle operation maps different parts of the compressed data for transmission to different nodes of the distributed data processing system; and

transmitting, in response to the data shuffle operation, the at least a portion of the compressed data from the first data shuffle memory pool to the second data shuffle memory pool in a peer-to-peer manner;

wherein the distributed data processing system is implemented via one or more processing devices.

2. The method of claim 1 , wherein the performance of the in-memory compression is triggered by a given policy.

3. The method of claim 2 , wherein the given policy is based on a memory pressure associated with a memory capacity threshold.

4. The method of claim 2 , wherein the given policy is based on a ratio of incoming write speed to outgoing read speed.

5. The method of claim 1 , wherein performing the in-memory compression further comprises:

obtaining one or more records;

compressing the obtained records into a work buffer;

copying the compressed records to an original location of the one or more compressed records; and

updating an active location, the active location indicating the beginning location of a next non-compressed record.

6. The method of claim 1 , wherein each of the first data shuffle memory pool and the second data shuffle memory pool comprise a hybrid memory configuration.

7. The method of claim 6 , wherein the hybrid memory configuration comprises a dynamic random access memory tier and one or more nonvolatile random access memory tiers.

8. The method of claim 7 , wherein the one or more nonvolatile random access memory tiers comprise a local nonvolatile random access memory tier and a remote nonvolatile random access memory tier.

9. The method of claim 7 , wherein the one or more nonvolatile random access memory tiers comprise a global nonvolatile random access memory tier.

10. The method of claim 1 , wherein each of the first data shuffle memory pool and the second data shuffle memory pool comprise a global shared memory tier.

11. The method of claim 1 , wherein data in the data set is characterized by the data shuffle writer node as one or more of: data to be transmitted to the data shuffle reader node; data to be infrequently locally accessed; and data to be sorted and aggregated as one or more extents.

12. The method of claim 1 , wherein the portion of the data set transmitted from the first data shuffle memory pool to the second data shuffle memory pool is transferred using remote direct memory access.

13. The method of claim 1 , wherein the portion of the data set to be transmitted is sorted in the data shuffle writer node based on one or more memory ranges associated with the data shuffle reader node.

14. The method of claim 1 , wherein the distributed data processing system comprises a MapReduce processing framework, and wherein the data shuffle writer node comprises a mapper node and the data shuffle read node comprises a reducer node.

15. A distributed data processing system, comprising:

a plurality of nodes wherein at least one node is configured as a data shuffle writer node and at least another node is configured as a data shuffle reader node, wherein:

the data shuffle writer node is configured to maintain a first data shuffle memory pool;

the data shuffle reader node is configured to maintain a second data shuffle memory pool; and further wherein:

the data shuffle writer node is configured to perform an in-memory compression on at least a portion of a data set from the first data shuffle memory pool;

the data shuffle writer node is configured to perform a data shuffle operation on at least a portion of the compressed data from the first shuffle memory pool, wherein the data shuffle operation maps different parts of the compressed data for transmission to different nodes of the distributed data processing system; and

the data shuffle writer node, in response to the data shuffle operation, is configured to transmit the at least a portion of the compressed data from the first data shuffle memory pool to the second data shuffle memory pool in a peer-to-peer manner;

wherein the distributed data processing system is implemented via one or more processing devices.

16. The system of claim 15 , wherein performance of the in-memory compression is triggered by a given policy, and wherein the given policy is based on a memory pressure associated with a memory capacity threshold or a ratio of incoming write speed to outgoing read speed.

17. The system of claim 15 , wherein each of the first data shuffle memory pool and the second data shuffle memory pool comprise a hybrid memory configuration.

18. The system of claim 15 , wherein the portion of the data set transmitted from the first data shuffle memory pool to the second data shuffle memory pool is transferred using remote direct memory access.

19. The system of claim 15 , wherein the distributed data processing system comprises a MapReduce processing framework, and wherein the data shuffle writer node comprises a mapper node and the data shuffle read node comprises a reducer node.

20. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by one or more processing devices of a distributed data processing system causes said one or more processing devices to:

maintain a first data shuffle memory pool at a data shuffle writer node and a second data shuffle memory pool at a data shuffle reader node, wherein the data shuffle writer node and the data shuffle reader node are part of a set of multiple nodes of the distributed data processing system;

perform an in-memory compression on at least a portion of a data set from the first data shuffle memory pool; and

perform a data shuffle operation on the at least a portion of the compressed data from the first shuffle memory pool, wherein the data shuffle operation maps different parts of the compressed data for transmission to different nodes of the distributed data processing system; and

transmit, in response to the data shuffle operation, the at least a portion of the compressed data from the first data shuffle memory pool to the second data shuffle memory pool in a peer-to-peer manner.