Method and system for enhancing a distributed storage system by decoupling computation and network tasks

1. A computer system, comprising:

a plurality of storage devices;

a first component configured to manage connections to the plurality of storage devices,

wherein a respective storage device comprises: a port coupled to the first component; at least one microprocessor; and a plurality of storage drives with non-volatile memory; and

a computing device which includes the first component,

wherein the computing device is configured to:

perform general computation tasks associated with the computer system;

receive a request to write data to a non-volatile memory of the computer system, wherein the first component is configured to process data to obtain processed data; and

transmit the processed data, via the first component, to a selected storage device; and

wherein the selected storage device is configured to write the processed data to a non-volatile memory of a first storage drive of the selected storage device by:

determining, by a flash translation layer module of the selected storage device, the first storage drive of the selected storage device to which to write the processed data;

performing, by a controller of the first storage drive, specific in-drive computation tasks on the processed data to obtain further processed data; and

writing, by the controller of the first storage drive, the further processed data to the non-volatile memory of the first storage drive.

2. The computer system of claim 1 , wherein the computing device further includes a central processing unit and a volatile memory.

3. The computer system of claim 2 , wherein the computing device comprises a single integrated circuit.

4. The computer system of claim 2 , wherein the central processing unit of the computing device is configured to perform one or more of:

managing meta-data;

placing data;

scrubbing, refreshing, or deleting data;

communicating with other computing devices via the first component; and

handling multiple replicas of data.

5. The computer system of claim 1 , further comprising:

a second component functioning as the network interface card, the access switch, and a central processing unit.

6. The computer system of claim 1 , wherein the first component is further configured to:

process data, including operations relating to a cyclic redundancy check, a hash function, and encryption;

handle network communication; and

handle communication between a pod switch and the plurality of storage devices.

7. The computer system of claim 1 ,

wherein the first component functions both as a network interface card and as an access switch, wherein the port comprises an Ethernet port, and wherein the respective storage device further comprises a plurality of PCIe lanes via which to access the plurality of storage drives,

wherein the at least one microprocessor and the PCIe lanes comprise a storage system on chip (SoC) component of the respective storage drive, and wherein the storage SoC component is configured to:

manage the plurality of storage drives; and

perform operations relating to a file system and a flash translation layer module associated with the storage drives.

8. The computer system of claim 1 , wherein a controller of a respective storage drive is configured to perform one or more of: erasure coding; compression; sorting; and filtering.

9. A computer-implemented method for facilitating data access in a computer system,

wherein the computer system comprises:

a plurality of storage devices;

a first component configured to manage connections to the plurality of storage devices,

wherein a respective storage device comprises: a port coupled to the first component; at least one microprocessor; and a plurality of storage drives with non-volatile memory; and

a computing device which includes the first component; and

wherein the method comprises:

receiving, by the computing device, a request to write data to a non-volatile memory of the computer system;

performing, by the computing device, general computation tasks associated with the computer system;

processing, by the first component, the data to obtain processed data;

transmitting the processed data, by the computing device via the first component, to a selected storage device; and

writing, by the selected storage device, the processed data to a non-volatile memory of a first storage drive of the selected storage device by:

determining, by a flash translation layer module of the selected storage device, the first storage drive of the selected storage device to which to write the processed data;

performing, by a controller of the first storage drive, specific in-drive computation tasks on the processed data to obtain further processed data; and

writing, by the controller of the first storage drive, the further processed data to the non-volatile memory of the first storage drive.

10. The method of claim 9 , wherein the computer system further comprises:

a computing device which includes a central processing unit and a volatile memory.

11. The method of claim 10 , wherein the computing device comprises a single integrated circuit.

12. The method of claim 10 , wherein the central processing unit of the computing device performs one or more of:

managing meta-data;

placing data;

scrubbing, refreshing, or deleting data;

communicating with other computing devices via the first component; and

handling multiple replicas of data.

13. The method of claim 9 , wherein the computer system further comprises:

a second component functioning as the network interface card, the access switch, and a central processing unit.

14. The method of claim 9 , further comprising:

processing data, by the first component, including operations relating to a cyclic redundancy check, a hash function, and encryption;

handling, by the first component, network communication; and

handling, by the first component, communication between a pod switch and the plurality of storage devices.

15. The method of claim 9 ,

wherein the first component functions both as a network interface card and as an access switch, wherein the port comprises an Ethernet port, and wherein the respective storage device further comprises a plurality of PCIe lanes via which to access the plurality of storage drives,

wherein the at least one microprocessor and the PCIe lanes comprise a storage system on chip (SoC) component of the respective storage drive, and wherein the method further comprises:

managing, by the storage SoC component, the plurality of storage drives; and

performing, by the storage SoC component, operations relating to a file system and a flash translation layer module.

16. The method of claim 9 , further comprising:

performing, by a controller of a respective storage drive, one or more of: erasure coding; compression; sorting; and filtering.

17. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method, wherein the computer comprises a storage device associated with a computer system,

wherein the computer system comprises:

a plurality of storage devices;

a first component configured to manage connections to the plurality of storage devices,

wherein a respective storage device comprises: a port coupled to the first component; at least one microprocessor; and a plurality of storage drives with non-volatile memory; and

a computing device which includes the first component; and

wherein the method comprises:

receiving, by the computing device, a request to write data to a non-volatile memory of the computer system;

performing, by the computing device, general computation tasks associated with the computer system;

processing, by the first component, the data to obtain processed data;

transmitting the processed data, by the computing device via the first component, to a selected storage device; and

writing, by the selected storage drive, the processed data to a non-volatile memory of a first storage drive of the selected storage device by:

determining, by a flash translation layer module of the selected storage device, the first storage drive of the selected storage device to which to write the processed data;

performing, by a controller of the first storage drive, specific in-drive computation tasks on the processed data to obtain further processed data; and

writing, by the controller of the first storage drive, the further processed data to the non-volatile memory of the first storage drive.