Data processing offload using in-storage code execution
Systems and methods for offloading data processing tasks using in-storage code execution are disclosed. For example, a data storage system including one or more processors, a non-transitory storage medium, and a storage logic executable by the one or more processors to perform operations including: receiving portable code configured to perform a data processing offload task, the portable code comprising one or more translatable, hardware-agnostic instructions for processing data stored on the storage medium; translating and executing the one or more translatable, hardware-agnostic instructions of the portable code to perform the data processing offload task using a translation and execution engine disposed in the storage logic; determining that use of the portable code has concluded; and responsive to determining that use of the portable code has concluded, releasing the portable code.
1. A data storage system, comprising:
a host device comprising:
one or more processors; and
a data processing offloader logic executable by the one or more processors to perform operations comprising:
creating a data processing procedure to perform a data processing offload task in a storage device configured to be coupled to the host device, wherein the storage device comprises:
a non-transitory storage medium; and
a storage controller configured to execute the data processing offload task using data stored in the non-transitory storage medium;
compiling the data processing procedure into a portable code configured to be loaded into the storage device, wherein the portable code comprises one or more translatable, hardware-agnostic instructions for processing data stored on the non-transitory storage medium of the storage device;
loading the portable code into the storage device for translation and execution of the portable code by the storage device;
detecting whether the execution of the portable code by the storage device has completed; and
responsive to detecting that the execution of the portable code has completed, unloading the portable code from the storage device.
2. The data storage system of claim 1 , wherein detecting whether the execution of the portable code by the storage device has completed comprises:
polling the storage device to periodically check a progress of the execution of the portable code by the storage device; and
receiving a status of the execution of the portable code from the storage device.
3. The data storage system of claim 1 , wherein responsive to detecting that the execution of the portable code has completed, receiving a result of the execution of the portable code from the storage device.
4. The data storage system of claim 1 , wherein loading the portable code into the storage device comprises generating an identifier uniquely identifying the loaded portable code.
5. The data storage system of claim 1 , wherein loading the portable code into the storage device comprises invoking the portable code previously stored in a precompiled format on the storage device.
6. The data storage system of claim 1 , wherein loading the portable code into the storage device comprises loading one or more identifiers uniquely identifying data to be processed on the non-transitory storage medium of the storage device.
7. The data storage system of claim 1 , wherein:
the data processing offloader logic skips compiling source code instructions of the data processing procedure; and
the one or more translatable, hardware-agnostic instructions of the portable code comprise the source code instructions.
8. The data storage system of claim 1 , wherein:
the data processing offloader logic compiles source code instructions of the data processing procedure into bytecode instructions;
the one or more translatable, hardware-agnostic instructions of the portable code comprise the bytecode instructions; and
the bytecode instructions are specifically configured to perform data processing offload tasks.
9. The data storage system of claim 1 , wherein the translation of the portable code by the storage device comprises compiling the one or more translatable, hardware-agnostic instructions of the portable code into machine code native to the storage device.
10. The data storage system of claim 1 , wherein the data processing offload task comprises:
performing one of a MapReduce, counting, filtering, compaction, and encryption type of operation on one or more items of data stored on the non-transitory storage medium of the storage device.
11. The data storage system of claim 1 , further comprising:
the storage device comprising:
one or more second processors;
the non-transitory storage medium; and
a storage controller logic executable by the one or more second processors to perform operations comprising:
receiving, from the host device, the portable code;
translating and executing the one or more translatable, hardware-agnostic instructions of the portable code to perform the data processing offload task in the storage device using a translation and execution engine disposed in the storage controller logic; and
sending to the host device an indication of a status of the data processing offload task in the storage device.
12. A method, comprising:
creating, by a host device, a data processing procedure to perform a data processing offload task in a storage device configured to be coupled to the host device, wherein the storage device comprises:
a non-transitory storage medium; and
a storage controller configured to execute the data processing offload task using data stored in the non-transitory storage medium;
compiling, by the host device, the data processing procedure into a portable code configured to be loaded into the storage device, wherein the portable code comprises one or more translatable, hardware-agnostic instructions for processing data stored on the non-transitory storage medium of the storage device;
loading, by the host device, the portable code into the storage device for translation and execution of the portable code by the storage device;
detecting, by the host device, whether the execution of the portable code by the storage device has completed; and
responsive to detecting that the execution of the portable code has completed, unloading, by the host device, the portable code from the storage device.
13. The method of claim 12 , wherein detecting whether the execution of the portable code by the storage device has completed comprises:
polling the storage device to periodically check a progress of the execution of the portable code by the storage device; and
receiving a status of the execution of the portable code from the storage device.
14. The method of claim 12 , wherein responsive to detecting that the execution of the portable code has completed, receiving a result of the execution of the portable code from the storage device.
15. The method of claim 12 , wherein loading the portable code into the storage device comprises generating an identifier uniquely identifying the loaded portable code.
16. The method of claim 12 , wherein loading the portable code into the storage device comprises invoking the portable code previously stored in a precompiled format on the storage device.
17. The method of claim 12 , wherein loading the portable code into the storage device comprises loading one or more identifiers uniquely identifying data to be processed on the non-transitory storage medium of the storage device.
18. A data storage system, comprising:
a host device;
a storage device configured to be coupled to the host device and including:
a non-transitory storage medium; and
a storage controller configured to execute a data processing offload task using data stored in the non-transitory storage medium;
means for creating a data processing procedure to perform the data processing offload task in the storage device;
means for compiling the data processing procedure into a portable code configured to be loaded into the storage device, the portable code comprising one or more translatable, hardware-agnostic instructions for processing data stored on the non-transitory storage medium of the storage device;
means for loading the portable code into the storage device for translation and execution of the portable code by the storage device;
means for detecting whether the execution of the portable code by the storage device has completed; and
means for unloading the portable code from the storage device responsive to detecting that the execution of the portable code has completed.