System and method for executing map-reduce tasks in a storage device
A system and method of providing enhanced data processing and analysis in an infrastructure for distributed computing and large-scale data processing. This infrastructure uses the Hadoop™ framework to divide an application into a large number of small fragments of work, each of which may be performed on one of a large number of compute nodes. The work may involve map tasks and reduce tasks which may be used to categorize and analyze large amounts of data in distributed systems. This infrastructure includes a cluster with a master node and a plurality of slave nodes. The slave nodes may include, or may be, intelligent solid-state drives capable of executing Map-Reduce functions. The use of intelligent solid-state drives reduces the need to exchange data with a CPU in a server.
1. An intelligent solid state drive comprising:
a solid state drive controller;
a solid state drive host interface; and
a flash memory;
the solid state drive host interface being selected from the group consisting of: PCIe, NVMe, SATA, Fibre Channel, SAS, SCSI over PCIe, and Ethernet,
the solid state drive controller configured to be in communication with the flash memory, and comprising:
a hardware engine; and
a microcontroller;
the solid state drive configured to:
receive, through the solid state drive host interface, a data query described by Map-Reduce functions;
execute map and reduce tasks, in the solid state drive controller, to process the query; and
return a query result through the solid state drive host interface.
2. The drive of claim 1 , wherein the intelligent solid state drive is configured to run an operating system.
3. The drive of claim 2 , wherein the operating system is configured to enable the drive to execute a high-level computer language.
4. The drive of claim 3 , wherein the computer language is an object-oriented programming language.
5. A distributed computing system, comprising a cluster of nodes,
the cluster of nodes comprising:
a master node; and
a plurality of slave nodes;
wherein a slave node of the plurality of slave nodes comprises a server comprising a server central processing unit (CPU) and the intelligent solid state drive of claim 1 .
6. A distributed computing system, comprising a cluster of nodes,
the cluster of nodes comprising:
a master node; and
a plurality of slave nodes;
wherein a slave node of the plurality of slave nodes is the intelligent solid state drive of claim 1 .
7. A method of executing a query in a distributed computing system comprising a master node and a plurality of slave nodes, a slave node of the plurality of slave nodes comprising the intelligent solid state drive of claim 1 , the method comprising:
submitting the query to the master node;
assigning a plurality of tasks to the plurality of slave nodes, by the master node, the plurality of tasks being configured to execute portions of the query;
executing the plurality of tasks, by the plurality of slave nodes;
returning the results of the execution of the plurality of tasks, by the plurality of slave nodes, to the master node; and
aggregating, by the master node, the results of the execution of the plurality of tasks.
8. The method of claim 7 , comprising assigning of a task by a first slave node of the plurality of slave nodes to a second slave node of the plurality of slave nodes.
9. The method of claim 7 , wherein a task of the plurality of tasks comprises an execution of a map function within a Map-Reduce framework.
10. The method of claim 7 , wherein a task of the plurality of tasks comprises an execution of a reduce function within a Map-Reduce framework.