Method and system for enhancing a distributed storage system by decoupling computation and network tasks

1. A computer system, comprising:

a plurality of storage devices; and

a first component of a computing device, wherein the first component is configured to function both as a network interface card and as an access switch, by handling communications to and from a plurality of pod switches and by handling communications to and from the plurality of storage devices, which allows the computing device to communicate with the pod switches without a separate layer for the access switch, wherein the first component is coupled to a port on a respective storage device,

wherein the respective storage device comprises: the port coupled to the first component; at least one microprocessor; a plurality of storage drives with non-volatile memory; and a plurality of PCIe lanes via which to access the plurality of storage drives, and

wherein the port comprises an Ethernet port, wherein the at least one microprocessor and the PCIe lanes comprise a storage system on chip (SoC) component of the respective storage drive, and wherein the storage SoC component is configured to:

manage the plurality of storage drives; and

perform operations relating to a file system and a flash translation layer module associated with the storage drives.

2. The computer system of claim 1 , further comprising:

wherein the computing device includes the first component, a central processing unit, and a volatile memory.

3. The computer system of claim 2 , wherein the computing device comprises a single integrated circuit.

4. The computer system of claim 2 , wherein the central processing unit of the computing device is configured to perform one or more of:

managing meta-data;

placing data;

scrubbing, refreshing, or deleting data;

communicating with other computing devices via the first component; and

handling multiple replicas of data.

5. The computer system of claim 2 ,

wherein the computing device is configured to:

receive a request to write data to a non-volatile memory of the computer system; and

transmit the data, via the first component, to a selected storage device; and

wherein the selected storage device is configured to:

write the data to a non-volatile memory of a first storage drive of the selected storage device.

6. The computer system of claim 5 ,

wherein the computing device is further configured to:

perform general computation tasks associated with the computer system;

process, by the first component, the data to obtain processed data; and

transmit the processed data, by the computing device via the first component, to the selected storage device.

7. The computer system of claim 6 ,

wherein in writing the data to the non-volatile memory of the first storage drive of the selected storage device, the selected storage device is further configured to:

determine, by a flash translation layer module of the selected storage device, the first storage drive of the selected storage device to which to write the processed data;

perform, by a controller of the first storage drive, specific in-drive computation tasks on the processed data to obtain further processed data; and

write, by the controller of the first storage drive, the further processed data to the non-volatile memory of the first storage drive.

8. The computer system of claim 1 , further comprising:

a second component functioning as the network interface card, the access switch, and a central processing unit.

9. The computer system of claim 1 , wherein the first component is further configured to:

process data, including operations relating to a cyclic redundancy check, a hash function, and encryption;

handle network communication; and

handle communication between a pod switch and the plurality of storage devices.

10. The computer system of claim 1 , wherein a controller of a respective storage drive is configured to perform one or more of: erasure coding; compression; sorting; and filtering.

11. A computer-implemented method for facilitating data access in a computer system,

wherein the computer system comprises:

a plurality of storage devices;

a first component of a computing device, wherein the first component is configured to function both as a network interface card and as an access switch, by handling communications to and from a plurality of pod switches and by handling communications to and from the plurality of storage devices, which allows the computing device to communicate with the pod switches without a separate layer for the access switch, wherein the first component is coupled to a port on a respective storage device,

wherein the respective storage device comprises: the port coupled to the first component; at least one microprocessor; a plurality of storage drives with non-volatile memory; and a plurality of PCIe lanes via which to access the plurality of storage drives, and

wherein the port comprises an Ethernet port, wherein the at least one microprocessor and the PCIe lanes comprise a storage system on chip (SoC) component of the respective storage drive; and

the computing device which includes the first component,

wherein the method comprises:

receiving, by the computing device, a request to write data to a non-volatile memory of the computer system;

transmitting the data, by the computing device via the first component, to a selected storage device;

writing the data to a non-volatile memory of a first storage drive of the selected storage device;

managing, by the storage SoC component, the plurality of storage drives; and

performing, by the storage SoC component, operations relating to a file system and a flash translation layer module.

12. The method of claim 11 , further comprising:

performing, by the computing device, general computation tasks associated with the computer system;

processing, by the first component, the data to obtain processed data; and

transmitting the processed data, by the computing device via the first component, to the selected storage device,

wherein writing the data to the non-volatile memory of the first storage drive of the selected storage device comprises:

determining, by a flash translation layer module of the selected storage device, the first storage drive of the selected storage device to which to write the processed data;

performing, by a controller of the first storage drive, specific in-drive computation tasks on the processed data to obtain further processed data; and

writing, by the controller of the first storage drive, the further processed data to the non-volatile memory of the first storage drive.

13. The method of claim 11 , wherein the computing device includes the first component, a central processing unit, and a volatile memory.

14. The method of claim 13 , wherein the computing device comprises a single integrated circuit.

15. The method of claim 13 , wherein the central processing unit of the computing device performs one or more of:

managing meta-data;

placing data;

scrubbing, refreshing, or deleting data;

communicating with other computing devices via the first component; and

handling multiple replicas of data.

16. The method of claim 11 , wherein the computer system further comprises:

a second component functioning as the network interface card, the access switch, and a central processing unit.

17. The method of claim 11 , further comprising:

processing data, by the first component, including operations relating to a cyclic redundancy check, a hash function, and encryption;

handling, by the first component, network communication; and

handling, by the first component, communication between a pod switch and the plurality of storage devices.

18. The method of claim 11 , further comprising:

performing, by a controller of a respective storage drive, one or more of: erasure coding; compression; sorting; and filtering.

19. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method, wherein the computer comprises a storage device associated with a computer system,

wherein the computer system comprises:

a plurality of storage devices;

a first component of a computing device, wherein the first component is configured to function both as a network interface card and as an access switch, by handling communications to and from a plurality of pod switches and by handling communications to and from the plurality of storage devices, which allows the computing device to communicate with the pod switches without a separate layer for the access switch, wherein the first component is coupled to a port on a respective storage device,

wherein the respective storage device comprises: the port coupled to the first component; at least one microprocessor; a plurality of storage drives with non-volatile memory; and a plurality of PCIe lanes via which to access the plurality of storage drives, and

wherein the port comprises an Ethernet port, wherein the at least one microprocessor and the PCIe lanes comprise a storage system on chip (SoC) component of the respective storage drive; and

the computing device which includes the first component,

wherein the method comprises:

receiving, by the computing device, a request to write data to a non-volatile memory of the computer system;

transmitting the data, by the computing device via the first component, to a selected storage device;

writing the data to a non-volatile memory of a first storage drive of the selected storage device;

managing, by the storage SoC component, the plurality of storage drives; and

performing, by the storage SoC component, operations relating to a file system and a flash translation layer module.

20. The storage medium of claim 19 , wherein the method further comprises:

performing, by the computing device, general computation tasks associated with the computer system;

processing, by the first component, the data to obtain processed data; and

transmitting the processed data, by the computing device via the first component, to the selected storage device,

wherein writing the data to the non-volatile memory of the first storage drive of the selected storage device comprises:

determining, by a flash translation layer module of the selected storage device, the first storage drive of the selected storage device to which to write the processed data;

performing, by a controller of the first storage drive, specific in-drive computation tasks on the processed data to obtain further processed data; and

writing, by the controller of the first storage drive, the further processed data to the non-volatile memory of the first storage drive.