Method and system for memory aware runtime to support multitenancy in heterogeneous clusters
View Patent ↗The invention solves the problem of sharing many-core devices (e.g. GPUs) among concurrent applications running on heterogeneous clusters. In particular, the invention provides transparent mapping of applications to many-core devices (that is, the user does not need to be aware of the many-core devices present in the cluster and of their utilization), time-sharing of many-core devices among applications also in the presence of conflicting memory requirements, and dynamic binding/binding of applications to/from many-core devices (that is, applications do not need to be statically mapped to the same many-core device for their whole life-time).
1. A system containing a heterogeneous cluster of many-core devices, wherein the many-core devices are physical hardware, said system comprising:
virtual-many-cores which execute concurrent applications, wherein the virtual-many-cores are dynamically bound to the many-core devices by a dispatcher, the virtual-many-cores allowing controlled time-sharing of the many-core devices;
a connection manager for receiving and enqueuing calls from the concurrent applications, wherein a call is either a memory allocation call, a data transfer call or a kernel call;
a memory manager which maintains memory allocation including a page table and swap area; and
the dispatcher dequeues the calls and based on whether the call is
a memory allocation call, then the dispatcher invokes the memory manager to allocate memory to an application from which the call originated, update the page table and returns a virtual pointer to the allocated memory to the application from which the call originated;
if not a memory allocation call whether the call is
a data transfer call, then the dispatcher invokes the memory
manager to update the page table and swap area;
if not a data transfer call whether the call is
a kernel call, then the dispatcher dynamically binds a virtual-many-core, associated with a concurrent application from which the call originated, to a many-core device of the many-core devices, wherein the memory manager de-allocates data used by the concurrent application from which the call originated but not required by the kernel call and allocates the data to another concurrent application.
2. The system of claim 1 , wherein when the dispatcher dynamically binds the virtual-many-core associated with a concurrent application from which the call originated to a many-core device of the many-core devices it favors idle many-core devices and prioritizes many-core devices with higher compute capabilities, and if all of the many core-devices are busy, the dispatcher evenly distributes the virtual-many-cores.