System and method for executing data processing tasks using resilient distributed datasets (RDDs) in a storage device
A system and method of providing enhanced data processing and analysis in an infrastructure for distributed computing and large-scale data processing. This infrastructure uses the Apache Spark framework to divide an application into a large number of small fragments of work, each of which may be performed on one of a large number of compute nodes. The work may involve Spark transformations, operations, and actions, which may be used to categorize and analyze large amounts of data in distributed systems. This infrastructure includes a cluster with a driver node and a plurality of worker nodes. The worker nodes may be, or may include, intelligent solid state drives capable of executing data processing functions under the Apache Spark framework. The use of intelligent solid state drives reduces the need to exchange data with a central processing unit (CPU) in a server.
1. An intelligent solid state drive comprising:
a storage interface;
a processing circuit;
a data buffer; and
a flash memory;
the processing circuit configured to be in communication with the flash memory, and comprising a central processing unit;
the intelligent solid state drive having a form factor selected from the group consisting of large form factor (LFF), small form factor (SFF), full-height, full-length (FH-FL) peripheral component interconnect express (PCIe) card, and full-height, half-length (FH-HL) peripheral component interconnect express (PCIe) card;
the intelligent solid state drive being configured to:
receive, through the storage interface, a first command comprising one or more transformations, and
retrieve file data from the flash memory and perform the one or more transformations on the file data to construct a portion of a resilient distributed dataset (RDD).
2. The intelligent solid state drive of claim 1 , wherein each of the one or more transformations is selected from the group consisting of map operations, filter operations, join operations, and combinations thereof.
3. The intelligent solid state drive of claim 1 , wherein the intelligent solid state drive is further configured, upon receiving the first command, to construct a lineage comprising a sequence of transformations corresponding to the first command.
4. The intelligent solid state drive of claim 3 , wherein the intelligent solid state drive is further configured to receive a second command requesting the elements of the portion of the RDD, and to perform the sequence of transformations of the lineage in response to receiving the second command.
5. The intelligent solid state drive of claim 4 , wherein the intelligent solid state drive is further configured, when the first command includes a cache action, to store at least some of the elements of the portion of the RDD in the data buffer.
6. The intelligent solid state drive of claim 3 , wherein the intelligent solid state drive is further configured, upon receipt of a second command, the second command including a collect operation, to transmit, through the storage interface, the elements of the portion of the RDD.
7. The intelligent solid state drive of claim 6 , wherein the intelligent solid state drive is further configured, upon receipt of the second command:
to store at least some of the elements of the portion of the RDD in the data buffer when the first command includes a cache action, and
to discard the elements of the portion of the RDD when the first command does not include a cache action.
8. The intelligent solid state drive of claim 3 , wherein the intelligent solid state drive is further configured, upon receipt of a second command, the second command including a persist action, to store at least some of the elements of the portion of the RDD in the data buffer.
9. The intelligent solid state drive of claim 1 , wherein the intelligent solid state drive is further configured to receive a broadcast variable as part of a closure, and to store it until instructed to delete it.
10. The intelligent solid state drive of claim 1 , wherein the intelligent solid state drive is further configured to receive an accumulator as a closure and to not perform any operations on the accumulator except to add to it.
11. The intelligent solid state drive of claim 1 , wherein the intelligent solid state drive is configured to run an operating system configured to enable the intelligent solid state drive to execute a high-level object-oriented programming language.
12. A distributed computing system, comprising a cluster of nodes,
the cluster of nodes comprising:
a driver node; and
a plurality of worker nodes;
wherein a worker node of the plurality of worker nodes is an intelligent solid state drive having a form factor selected from the group consisting of large form factor (LFF), small form factor (SFF), full-height, full-length (FH-FL) peripheral component interconnect express (PCIe) card, and full-height, half-length (FH-HL) peripheral component interconnect express (PCIe) card.
13. A method of executing a data processing task in a distributed computing system comprising a driver node and a plurality of worker nodes,
a worker node of the plurality of worker nodes comprising an intelligent solid state drive,
the method comprising:
submitting the data processing task to the driver node; assigning a plurality of tasks to the plurality of worker nodes, by the driver node, the plurality of tasks being configured to execute portions of the data processing task;
executing the plurality of tasks, by the plurality of worker nodes;
returning the results of the execution of the plurality of tasks, by the plurality of worker nodes, to the driver node; and
aggregating, by the driver node, the results of the execution of the plurality of tasks,
the intelligent solid state drive having a form factor selected from the group consisting of large form factor (LFF), small form factor (SFF), full-height, full-length (FHFL) peripheral component interconnect express (PCIe) card, and full-height, half-length (FH-HL) peripheral component interconnect express (PCIe) card.
14. The method of claim 13 , comprising assigning of a task by a first worker node of the plurality of worker nodes to a second worker node of the plurality of worker nodes.
15. The method of claim 13 , wherein a task of the plurality of tasks comprises an execution of a machine learning function within an Apache Spark framework.
16. The method of claim 13 , wherein a task of the plurality of tasks comprises an execution of reduce and collect operations within an Apache Spark framework.