Firmware-controlled memory binding and implicit data persistence for hardware accelerators
Various examples, systems, and methods relate to facilitating memory use by a hardware accelerator. A system can check whether an on-chip memory region has already been set up for a given use. If it has, the system can reuse it. If not, the system can allocate a new region and connect it to a larger off-chip memory. The system can determine whether to load data into on-chip memory or save data back to off-chip memory based on various rules.
1 . A system, comprising:
one or more processors to:
allocate, via a firmware interface, a local memory region in a local memory of a hardware accelerator bound to an external memory region, the allocation returning a device pointer corresponding to a binding, the binding tracked using a resource identifier in a resource table corresponding with a non-privileged execution context; and
provide the device pointer to a memory region accessible to the non-privileged execution context for at least one subsequent construction of at least one command buffer identifying the binding of the local memory region to the external memory region.
2 . The system of claim 1 , wherein the one or more processors are to:
construct, in the external memory region of the hardware accelerator, the at least one command buffer comprising at least one command identifying the device pointer corresponding to the binding of the local memory region to the external memory region.
3 . The system of claim 2 , wherein the one or more processors are to:
cause execution of the at least one command using the external memory region via the local memory region by a direct memory access (DMA) controller using the at least one command buffer, wherein the at least one command corresponds to at least one of (i) reading data from the external memory region into the local memory region, or (ii) writing data from the local memory region to the external memory region.
4 . The system of claim 1 , wherein allocating comprises:
providing, via the firmware interface, a request to allocate the local memory region in the local memory of the hardware accelerator and bind the local memory region to the external memory region; and
receiving, via the firmware interface, the device pointer corresponding to the binding of the local memory region to the external memory region;
wherein the request is generated by a privileged execution context as a firmware-level configuration command using a firmware access application programming interface (API), and the firmware-level configuration command identifies the binding of the local memory region to the external memory region.
5 . The system of claim 1 , wherein the at least one command buffer comprises descriptor data identifying the device pointer, and wherein the descriptor data is updated, prior to execution, to identify a memory access path from the external memory region via the local memory region, and wherein the memory access path corresponds to a patched physical address in the local memory region identified for the execution of at least one command.
6 . The system of claim 1 , wherein the device pointer corresponds to an address indirection comprising the binding of the local memory region to the external memory region, the address indirection identified by a DMA controller to perform at least one memory access command during execution.
7 . The system of claim 1 , wherein the one or more processors are to:
construct or receive, in the external memory region of the hardware accelerator, a second command buffer comprising at least one second command identifying a resource corresponding to the local memory region;
provide, to a submission queue corresponding with the non-privileged execution context, submission information for the second command buffer;
provide, to a signaling interface corresponding with the submission queue, a submission indicator corresponding to the submission information; and
cause execution of the second command buffer by at least one hardware component of the hardware accelerator.
8 . The system of claim 1 , wherein the one or more processors are to:
prior to allocating the local memory region, determine whether a previously allocated local memory region bound to the external memory region exists for the non-privileged execution context;
in response to determining that the previously allocated local memory region exists, increment a reference count corresponding to the binding; and
in response to determining that the previously allocated local memory region does not exist, transmit, to the firmware interface, a request to allocate the local memory region in the local memory of the hardware accelerator bound to the external memory region in accordance with at least one access policy.
9 . The system of claim 8 , wherein the one or more processors are to:
decrement the reference count corresponding to the binding in response to a release request; and
release the local memory region in response to determining that the reference count satisfies a release condition.
10 . The system of claim 1 , wherein the one or more processors are to:
initiate a flush operation to transfer data from the local memory region to the external memory region in response to a flush request, wherein the flush operation occurs in response to a user context switch; or
initiate an invalidate operation marking data stored in the local memory region as invalid in response to an invalidate request to trigger reinitialization of the local memory region during a subsequent access;
wherein initialization of the local memory region with data from the external memory region and flushing of the local memory region to the external memory region occur in response to the user context switch, and wherein the flushing is further performable in response to an explicit request.
11 . The system of claim 1 , wherein the binding is established in accordance with at least one access policy, the at least one access policy comprising a first policy to initialize the local memory region with data from the external memory region or a second policy to flush data from the local memory region to the external memory region during release.
12 . The system of claim 1 , wherein the firmware interface maintains at least two concurrent bindings of a plurality of local memory regions to corresponding a plurality of external memory regions for the non-privileged execution context.
13 . The system of claim 1 , wherein the firmware interface comprises a compatibility layer configured to translate memory allocation and at least one access request from at least one software application implemented for a version of the hardware accelerator, the compatibility layer updating at least one request to invoke the allocation of the local memory region and the binding to the external memory region via the firmware interface.
14 . The system of claim 1 , wherein the at least one command buffer comprises descriptor data identifying the device pointer, and wherein the descriptor data is modified, by firmware, to identify a source address and a destination address for a DMA transaction by a DMA controller, the source address or destination address corresponding to the local memory region and the external memory region bound by the device pointer, the descriptor data comprising a transfer size and at least one access policy.
15 . The system of claim 1 , wherein the one or more processors are to execute operations comprising:
a system for managing memory persistence using firmware-level resource tracking;
a system for binding one or more local memory regions to one or more DRAM-backed regions via the firmware interface;
a system for allocating one or more memory regions using one or more access policies;
a system for inserting one or more flush or invalidate operations in accordance with DRAM persistence;
a system for storing one or more device pointers for command buffer reuse;
a system implementing one or more multi-model language models (MMLMs);
a system implementing one or more large language models (LLMs);
a system implementing one or more small language models (SLMs);
a system implementing one or more vision language models (VLMs);
a system for generating synthetic data;
a system for generating synthetic data using AI;
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing remote operations;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
16 . A system-on-a-chip (SoC), comprising:
processing circuitry to:
transmit, via a firmware interface, a request to allocate a local memory region in a local memory of a hardware accelerator and bind the local memory region to an external memory region;
receive, via the firmware interface, a device pointer corresponding to the binding of the local memory region to the external memory region, the binding tracked using a resource identifier in a resource table corresponding with a non-privileged execution context; and
store the device pointer in a memory region accessible to a non-privileged execution context for at least one subsequent construction of at least one command buffer identifying the binding of the local memory region to the external memory region.
17 . A method, comprising:
allocating, by one or more processors via a firmware interface, a local memory region in a local memory of a hardware accelerator bound to an external memory region, the allocation returning a device pointer corresponding to a binding, the binding tracked using a resource identifier in a resource table corresponding with a non-privileged execution context; and
providing, by the one or more processors, the device pointer to a memory region accessible to the non-privileged execution context for at least one subsequent construction of at least one command buffer identifying the binding of the local memory region to the external memory region.
18 . The method of claim 17 , further comprising:
constructing, by the one or more processors in the external memory region of the hardware accelerator, the at least one command buffer comprising at least one command identifying the device pointer corresponding to the binding of the local memory region to the external memory region.
19 . The method of claim 18 , further comprising:
causing, by the one or more processors, execution of the at least one command using the external memory region via the local memory region by a direct memory access (DMA) controller using the at least one command buffer, wherein the at least one command corresponds to at least one of (i) reading data from the external memory region into the local memory region, or (ii) writing data from the local memory region to the external memory region.
20 . The method of claim 17 , wherein allocating comprises:
providing, via the firmware interface, a request to allocate the local memory region in the local memory of the hardware accelerator and bind the local memory region to the external memory region; and
receiving, via the firmware interface, the device pointer corresponding to the binding of the local memory region to the external memory region;
wherein the request is generated by a privileged execution context as a firmware-level configuration command using a firmware access application programming interface (API), and the firmware-level configuration command identifies the binding of the local memory region to the external memory region.