Storage edge controller with a metadata computational engine

1. A storage device controller for managing storage and retrieval of data at one or more storage devices, the storage device controller comprising:

a host interface configured to communicate with one or more hosts;

a memory interface, configured to communicate locally with a non-volatile memory of the one or more storage devices;

a first processor for managing local storage or retrieval of data objects at the non-volatile memory, the first processor being configured to:

determine whether the first processor is in a low utilization state for controlling storage operations; and

based on the determination, selectively store in the non-volatile memory received data objects that are received from the one or more hosts, or retrieve from the non-volatile memory stored data objects that are stored in the non-volatile memory to be provided to a computational engine, including retrieving stored data objects from the non-volatile memory in response to determining that the first processor is in the low utilization state;

wherein the computational engine is configured to:

obtain, from a volatile memory, a first computational model and a set of parameters for implementing the first computational model for computing metadata that defines content characteristics of the data objects, and

using the first computational model, compute the metadata of the stored data objects that are selectively retrieved from the non-volatile memory or compute the metadata of selected received data objects that are received from the one or more hosts for storage in the non-volatile memory before the received data objects are stored by the first processor in the non-volatile memory.

2. The storage device controller of claim 1 , wherein the volatile memory is a dynamic random-access memory coupled to the storage device controller.

3. The storage device controller of claim 1 , wherein the volatile memory is a host memory buffer allocated by a host system to the storage device controller, and wherein the host memory buffer is accessible by the storage device controller over a computer network connection, or a bus connection.

4. The storage device controller of claim 1 , wherein the computational engine comprises:

a second processor that is separate from the first processor and is configured to perform computational tasks relating to metadata generation including implementing the first computational model, and

wherein the first processor is configured to:

send a computational task relating to metadata generation to the second processor at the computational engine without taking up resource of the first processor for an ongoing operation being performed by the storage device controller.

5. The storage device controller of claim 4 , wherein the computational engine further comprises:

a volatile memory coupled to the second processor, wherein the volatile memory is a static random-access memory configured to cache at least a portion of the data objects during computation of the metadata that defines content characteristics of the cached portion of data objects.

6. The storage device controller of claim 1 , wherein the first processor is further configured to:

receive, via the host interface, the data objects from the one or more hosts, wherein the data objects are to be stored at the non-volatile memory;

temporarily store the received data objects at a volatile memory disposed within the storage device controller for metadata computation;

after computation of the metadata that defines content characteristics of the data objects is completed:

send, via the memory interface, the received data objects from the volatile memory to the non-volatile memory for storage; and

perform at least one of: sending, via the host interface, the metadata to the host system, and sending, via the memory interface, the metadata to the non-volatile memory for storage.

7. The storage device controller of claim 1 , wherein the first processor is further configured to:

receive, via the host interface, a command from a host system of the one or more hosts to retrieve the data objects from the non-volatile memory;

in response to the command, retrieve, via the memory interface, the data objects from the volatile memory disposed within the storage controller for metadata computation; and

after computation of metadata defining content characteristics of the data objects is completed, send, via the memory interface, the metadata and the data objects to the non-volatile memory for storage.

8. The storage device controller of claim 1 , wherein the first processor is further configured to:

receive, via the host interface and from the one or more hosts, a request for metadata while the computational engine is computing the metadata; and

respond to the request for metadata asynchronously by:

waiting until the requested metadata is computed at the computational engine; and

sending, via the host interface, the requested metadata to the host system while new metadata, different from the requested metadata, is being computed at the computational engine.

9. The storage device controller of claim 7 , wherein the first processor is further configured to:

in response to the command, determine whether the command from the host system requires an update of the first computational model;

in response to determining that the command from the host system does not require an update of the computational model, instruct the computational engine to implement an existing computational model;

in response to determining that the command from the host system requires the first computational model to be updated to a second computational model different from the first computational model:

retrieve a set of updated parameters for the second computational model from the volatile memory disposed within the storage controller, or, via the host interface, from a host buffer memory disposed within the host system, and

send the set of updated parameters to the computational engine to implement the second computational model.

10. The storage device controller of claim 1 , wherein the computational engine is further configured to automatically generate metadata that defines content characteristics of the objects by performing any of identification of persons of interest or other objects; customized insertion of advertisements into streamed videos; cloud-based analytics of data from autonomous vehicles; analytics of call and response quality in a ChatBot Voice calls database, text documents and text messages database analysis; mood detection; scene identification within a video file or voice call; identification of persons or objects in surveillance footage; identification of types of actions occurring in surveillance footage; identification of voices or types of sounds in recordings; classification of phrases and responses used during conversations; and analysis of automotive sensor data and driving response.

11. A method for managing storage and retrieval of data at one or more storage devices, the method comprising:

communicating, via a host interface of a storage device controller, with one or more hosts;

communicating, via a memory interface of the storage device controller, locally with a non-volatile memory of the one or more storage devices;

determine whether a first processor of the storage device controller is in a low utilization state for controlling storage operations;

managing, via the first processor of the storage device controller, local storage or retrieval of data objects at the non-volatile memory, wherein the first processor is configured to, based on the determination, selectively store in the non-volatile memory received data objects that are received from the one or more hosts, or retrieve from the non-volatile memory stored data objects that are stored in the non-volatile memory to be provided to a computational engine, including retrieving stored data objects from the non-volatile memory in response to determining that the first processor is in the low utilization state;

obtaining, from a volatile memory, a first computational model and a set of parameters to implement the first computational model to compute the metadata that defines content characteristics of the data objects; and

computing, via the computational engine using the computational model, the metadata of the stored data objects that are selectively retrieved from the non-volatile memory or the metadata of selected received data objects that are received from the one or more hosts for storage in the non-volatile memory before the received data objects are stored by the first processor in the non-volatile memory.

12. The method of claim 11 , further comprising:

storing parameters of the first computational model for the computational engine at the volatile memory,

wherein the volatile memory is a dynamic random-access memory coupled to the storage device controller.

13. The method of claim 11 , further comprising:

storing parameters of the computational model for the computational engine at the volatile memory, wherein the volatile memory is a host memory buffer allocated by a host system to the storage device controller, and wherein the host memory buffer is accessible by the storage device controller over a computer network connection, or a bus connection.

14. The method of claim 11 , further comprising:

implementing the computational model by a second processor that is separate from the first processor; and

sending, by the first processor, a computational task relating to metadata generation to the second processor at the computational engine without taking up resource of the first processor for an ongoing operation being performed by the storage device controller.

15. The method of claim 14 , further comprising:

caching, at least a portion of the data objects during computation of the metadata that defines content characteristics of the cached portion of data objects, at a volatile memory coupled to the second processor, wherein the volatile memory is a static random-access memory.

16. The method of claim 11 , further comprising:

receiving, via the host interface, the data objects from the one or more hosts, wherein the objects are to be stored at the non-volatile memory;

temporarily storing the received data objects at a volatile memory disposed within the storage device controller for metadata computation;

after computation of the metadata that defines content characteristics of the data objects is completed:

sending, via the memory interface, the received data objects from the volatile memory to the non-volatile memory for storage; and

performing at least one of: sending, via the host interface, the metadata to the host system, and sending, via the memory interface, the metadata to the non-volatile memory for storage.

17. The method of claim 11 , further comprising:

receiving, via the host interface, a command from a host system of the one or more hosts to retrieve the objects from the non-volatile memory;

in response to the command, retrieving, via the memory interface, the data objects from the volatile memory to a volatile memory disposed within the storage controller for metadata computation; and

after computation of the metadata that defines content characteristics of the data objects is completed, sending, via the memory interface, the metadata and the data objects to the non-volatile memory for storage.

18. The method of claim 11 , further comprising:

receiving, via the host interface and from the one or more hosts, a request for metadata while the computational engine is computing the metadata; and

responding to the request for metadata asynchronously by:

waiting until the requested metadata is computed at the computational engine; and

sending, via the host interface, the requested metadata to the host system while new metadata different from the requested metadata is being computed at the computational engine.

19. The method of claim 17 , further comprising:

in response to the command, determining whether the command requires an update of the first computational model;

in response to determining that the command from the host system requires no update of the first computational model, instructing the computational engine to implement the first computational model;

in response to determining that the command from the host system requires the first computational model to be updated to a second computational model different from the first computational model:

retrieving a set of updated parameters for the second computational model from the volatile memory disposed within the storage controller, or via the host interface and from a host buffer memory disposed within the host system, and

sending the set of updated parameters to the computational engine to implement the second computational model.

20. The method of claim 11 , further comprising:

generating metadata that defines content characteristics of the objects by automatically performing any of identification of persons of interest or other objects; customized insertion of advertisements into streamed videos; cloud-based analytics of data from autonomous vehicles; analytics of call and response quality in a ChatBot Voice calls database, text documents and text messages database analysis; mood detection; scene identification within a video file or voice call; identification of persons or objects in surveillance footage; identification of types of actions occurring in surveillance footage; identification of voices or types of sounds in recordings; classification of phrases and responses used during conversations; and analysis of automotive sensor data and driving response.