Methods and systems for parallel distributed computation

1. A machine implemented method, comprising:

configuring a shared computing task among a plurality of computing nodes each computing node executing an application for performing the shared computing task;

generating intermediate data by each of the plurality of computing nodes during a mapping process for performing the shared computing task;

capturing the intermediate data for each of the plurality of computing nodes and transferring the intermediate data of each of the plurality of computing nodes to a shared storage device;

storing the intermediate data for each of the plurality of computing nodes at a key-value data store at the shared storage device that enables searching for the intermediate data of any node at the shared storage device using a key-value pair; and

using the key-value pair to obtain the intermediate data from the shared storage device when needed for a reduce process of the shared computing task; wherein the mapping process and the reduce process are executed by a processor coupled to a memory device.

2. The method of claim 1 , wherein a filter driver executed by each of the computing node interfaces with the mapping process to capture the intermediate data generated by its node and sends the intermediate data to a message module for sending it to the shared storage device.

3. The method of claim 2 , wherein a shuffle plugin module executed by the processor obtains the intermediate data from the shared storage using the key-value pair and provides the intermediate data to the reduce process, when needed.

4. The method of claim 1 , wherein the plurality of computing nodes operate within a Hadoop architecture for executing the shared computing task.

5. The method of claim 1 , wherein each computing node interfaces with a converged network adapter that processes both network and storage traffic to transfer the intermediate data to the shared storage.

6. The method of claim 1 , wherein the shared storage is managed by an intelligent storage adapter that can access the shared storage and other storage via a storage area network.

7. The method of claim 6 , wherein the intelligent storage adapter has access to a solid state storage device for storing the intermediate data.

8. A non-transitory, machine readable storage medium having stored thereon instructions for performing a method, comprising machine executable code which when executed by at least one machine, causes the machine to:

configure a shared computing task among a plurality of computing nodes each computing node executes an application for performing the shared computing task;

generate intermediate data by each of the plurality of computing nodes during a mapping process for performing the shared computing task;

capture the intermediate data for each of the plurality of computing nodes and transfer the intermediate data of each of the plurality of computing nodes to a shared storage device;

store the intermediate data for each of the plurality of computing nodes at a key-value data store at the shared storage device that enables searching for the intermediate data of any node at the shared storage device using a key-value pair; and

use the key-value pair to obtain the intermediate data from the shared storage device when needed for a reduce process of the shared computing task; wherein the mapping process and the reduce process are executed by a processor coupled to a memory device.

9. The non-transitory, storage medium of claim 8 , wherein a filter driver executed by each of the computing node interfaces with the mapping process to capture the intermediate data and sends the intermediate data to a message module for sending it to the shared storage device.

10. The non-transitory, storage medium of claim 9 , wherein a shuffle plugin module executed by the processor obtains the intermediate data from the shared storage using the key-value pair and provides the intermediate data to the reduce process, when needed.

11. The non-transitory, storage medium of claim 8 , wherein the plurality of computing nodes operate within a Hadoop architecture for executing the shared computing task.

12. The non-transitory, storage medium of claim 8 , wherein each computing node interfaces with a converged network adapter that processes both network and storage traffic to transfer the intermediate data to the shared storage.

13. The non-transitory, storage medium of claim 8 , wherein the shared storage is managed by an intelligent storage adapter that can access the shared storage and other storage via a storage area network.

14. The non-transitory, storage medium of claim 13 , wherein the intelligent storage adapter has access to a solid state storage device for storing the intermediate data.

15. A system, comprising:

a memory containing machine readable medium comprising machine executable code having stored thereon instructions; and

a processor module coupled to the memory, the processor module configured to execute the machine executable code to:

configure a shared computing task among a plurality of computing nodes each computing node executes an application for performing the shared computing task;

generate intermediate data by each of the plurality of computing nodes during a mapping process for performing the shared computing task;

capture the intermediate data for each of the plurality of computing nodes and transfer the intermediate data of each of the plurality of computing nodes to a shared storage device;

store the intermediate data for each of the plurality of computing nodes at a key-value data store at the shared storage device that enables searching for the intermediate data of any node at the shared storage device using a key-value pair; and

use the key-value pair to obtain the intermediate data from the shared storage device when needed for a reduce process of the shared computing task; wherein the mapping process and the reduce process are executed by a processor coupled to a memory device.

16. The system of claim 15 , wherein a filter driver executed by each of the computing node interfaces with the mapping process to capture the intermediate data and sends the intermediate data to a message module for sending it to the shared storage device.

17. The system of claim 16 , wherein a shuffle plugin module executed by the processor obtains the intermediate data from the shared storage using the key-value pair and provides the intermediate data to the reduce process, when needed.

18. The system of claim 15 , wherein the plurality of computing nodes operate within a Hadoop architecture for executing the shared computing task.

19. The system of claim 15 , wherein each computing node interfaces with a converged network adapter that processes both network and storage traffic to transfer the intermediate data to the shared storage.

20. The system of claim 15 , wherein the shared storage is managed by an intelligent storage adapter that can access the shared storage and other storage via a storage area network.