Function processing using storage controllers for load sharing
Systems and methods for distributed storage and processing systems using storage controllers for load sharing are described. A host processor may receive a function request that corresponds to a plurality of compute tasks, such as map compute tasks targeting data in local storage. The host processor may fetch the data from local storage devices through storage controllers. At least one storage controller, such as a non-volatile memory express (NVMe) interface controller, may be configured to execute overflow tasks for the function request. Another storage controller may be configured for executing other processing and management activities, such as reduce compute tasks.
1. A system, comprising:
a host processor configured to:
receive a function request for processing data from at least one storage device, the function request including a plurality of compute tasks;
fetch, from the at least one storage device, input data for a first subset of the plurality of compute tasks;
execute the first subset of the plurality of compute tasks; and
return, based on the plurality of compute tasks, a function result; and
a first storage controller comprising:
a first controller processor configured to execute a second subset of the plurality of compute tasks; and
a storage device interface to a plurality of storage devices, including the at least one storage device, wherein the first storage controller is configured to:
fetch, from the at least one storage device, the input data for the plurality of compute tasks through the storage device interface;
send, to the host processor, the input data for the first subset of the plurality of compute tasks; and
process the input data for the second subset of the plurality of compute tasks.
2. The system of claim 1 , wherein the first storage controller is further configured to:
receive a requested processing time; and
determine the second subset of the plurality of compute tasks based on:
excluding the first subset of the plurality of compute tasks; and
a requested processing time for the function request.
3. The system of claim 2 , wherein the host processor is further configured to:
receive, with the function request, the requested processing time;
determine, based on the requested processing time, the first subset of the plurality of compute tasks; and
send, to the first storage controller:
the requested processing time; and
the first subset of the plurality of compute tasks.
4. The system of claim 1 , wherein each of the plurality of compute tasks operates on a data unit of the input data from a first namespace.
5. The system of claim 1 , wherein:
the first storage controller is further configured to:
fetch the input data from a first namespace; and
store, responsive to executing the second subset of the plurality of compute tasks, overflow task intermediate data in a second namespace; and
the host processor is further configured to store, responsive to executing the first subset of the plurality of compute tasks, host task intermediate data in the second namespace.
6. The system of claim 1 , further comprising:
a second storage controller configured to:
provide access to the at least one storage device for the host processor;
fetch function intermediate data generated from the plurality of compute tasks;
execute, using the function intermediate data, at least one reduce compute task to generate function output data; and
store the function output data in the at least one storage device, wherein the function result returned by the host processor is based on the function output data.
7. The system of claim 6 , wherein:
the first storage controller is further configured to store, responsive to executing the second subset of the plurality of compute tasks, overflow task intermediate data in a second namespace;
the host processor is further configured to store, responsive to executing the first subset of the plurality of compute tasks, host task intermediate data in the second namespace; and
the function intermediate data includes:
the overflow task intermediate data; and
the host task intermediate data.
8. The system of claim 7 , wherein:
the second storage controller is further configured to:
store the function output data in a third namespace; and
send an output data notification to the host processor; and
the host processor is further configured to return, responsive to the output data notification, the function result using the function output data in the third namespace.
9. The system of claim 6 , wherein:
executing the first of the plurality of compute tasks generates host task intermediate data;
the host processor is further configured to:
evaluate a size of the host task intermediate data against a size threshold;
selectively store, responsive to the size threshold being met, the host task intermediate data to the at least one storage device; and
selectively send, responsive to the size threshold not being met, the host task intermediate data to the second storage controller; and
the function intermediate data includes the host task intermediate data.
10. The system of claim 6 , wherein:
the first storage controller is configured as a master non-volatile memory express controller for accessing a shared namespace in the at least one storage device;
the second storage controller is configured as a secondary non-volatile memory express controller for accessing the shared namespace in the at least one storage device;
the plurality of compute tasks are map tasks for a map-reduce function;
each of the plurality of compute tasks operates on a data unit of the input data from the shared namespace; and
the second storage controller is further configured to send, responsive to receiving intermediate data for each compute task of the plurality of compute tasks, a progress notification to the host processor.
11. A computer-based method, comprising:
receiving, at a host processor, a function request for processing data from at least one storage device, the function request including a plurality of compute tasks;
accessing, though a first storage controller, at least one storage device;
fetching, by the host processor and from a first namespace in the at least one storage device, input data for a first subset of the plurality of compute tasks;
executing, by the host processor, the first subset of the plurality of compute tasks;
storing, responsive to executing the first subset of the plurality of compute tasks, host task intermediate data in a second namespace in the at least one storage device;
fetching, to the first storage controller and from the first namespace, input data for a second subset of the plurality of compute tasks;
executing, by the first storage controller, the second subset of the plurality of compute tasks;
storing, responsive to executing the second subset of the plurality of compute tasks, overflow task intermediate data in the second namespace; and
returning, based on the host task intermediate data and the overflow task intermediate data, a function result.
12. The computer-based method of claim 11 , further comprising:
receiving a requested processing time for the function request; and
determining the second subset of the plurality of compute tasks based on:
excluding the first subset of the plurality of compute tasks; and
the requested processing time for the function request.
13. The computer-based method of claim 12 , further comprising:
determining, based on the requested processing time, the first subset of the plurality of compute tasks.
14. The computer-based method of claim 11 , further comprising:
fetching, from the at least one storage device and to the first storage controller, the input data for the plurality of compute tasks; and
sending, from the first storage controller to the host processor, the input data for the first subset of the plurality of compute tasks.
15. The computer-based method of claim 11 , further comprising:
executing, using the host task intermediate data and the overflow task intermediate data, at least one reduce compute task to generate the function result.
16. The computer-based method of claim 11 , further comprising:
providing, from a second storage controller, access to the at least one storage device for the host processor;
fetching, by the second storage controller, function intermediate data generated from the plurality of compute tasks;
executing, by the second storage controller and using the function intermediate data, at least one reduce compute task to generate function output data; and
storing the function output data in the at least one storage device, wherein the function result returned is based on the function output data.
17. The computer-based method of claim 16 ,
wherein the function intermediate data includes:
the overflow task intermediate data; and
the host task intermediate data.
18. The computer-based method of claim 17 , further comprising:
storing the function output data in a third namespace;
sending, by the second storage controller, an output data notification to the host processor; and
returning, responsive to the output data notification, the function result using the function output data in the third namespace.
19. The computer-based method of claim 16 :
wherein
the function intermediate data includes the host task intermediate data; and
further comprising:
evaluating a size of the host task intermediate data against a size threshold;
selectively storing, responsive to the size threshold being met, the host task intermediate data to the at least one storage device; and
selectively sending, responsive to the size threshold not being met, the host task intermediate data to the second storage controller.
20. A system, comprising:
a host processor;
a plurality of storage devices;
a first storage controller configured for the host processor to access data on the plurality of storage devices;
a second storage controller configured for the host processor to access data on the plurality of storage devices;
means for receiving, at the host processor, a function request for processing data from at least one storage device, the function request including a plurality of compute tasks;
means for fetching, by the host processor and from the at least one storage device, input data for a first subset of the plurality of compute tasks;
means for executing, by the host processor, the first subset of the plurality of compute tasks;
means for executing, by the first storage controller, a second subset of the plurality of compute tasks;
means for fetching, by the second storage controller, function intermediate data generated from the plurality of compute tasks;
means for executing, by the second storage controller and using the function intermediate data, at least one reduce compute task to generate function output data;
means for storing the function output data in the at least one storage device; and
means for returning a function result based on the function output data.