System cache optimizations for deep learning compute engines
In an example, an apparatus comprises a plurality of compute engines; and logic, at least partially including hardware logic, to detect a cache line conflict in a last-level cache (LLC) communicatively coupled to the plurality of compute engines; and implement context-based eviction policy to determine a cache way in the cache to evict in order to resolve the cache line conflict. Other embodiments are also disclosed and claimed.
1. An apparatus comprising:
a plurality of compute engines communicatively coupled to a last-level cache (LLC) by an interconnect, the LLC configured with a variable cache line size, wherein the plurality of compute engines execute one or more layers of a deep learning (DL) network using the LLC; and
a controller to:
assign one or more of the plurality of compute engines as clients of the LLC;
assign a context identifier (ID) to the clients of the LLC;
receive, from the plurality of compute engines, a plurality of cache access requests, each cache access request comprising:
the context ID corresponding to the compute engine which originated the cache access request; and
metadata that indicates a total size of data to be accessed in one or more subsequent data access transactions;
detect a cache line conflict in the LLC between a first cache access request from a first compute engine and a second cache access request from a second compute engine;
responsive to detecting the cache line conflict, implement a context-based cache eviction policy that utilizes the context ID of the first compute engine and the context ID of the second compute engine to determine a cache way in the LLC to evict in order to resolve the cache line conflict, wherein the LLC is dynamically configured with partitions corresponding to each of the clients based on the one or more layers of the DL network that each of the clients execute; and
responsive to a cache miss, select a variable cache line size to allocate and fill from the LLC, wherein the variable cache line size is selected based on the total size of the data indicated in the metadata received with the access request, and based on data utilization of the LLC by the DL network.
2. The apparatus of claim 1 , wherein:
the cache eviction policy is a function of the context identifier.
3. The apparatus of claim 1 , wherein:
the LLC is reconfigured dynamically into a plurality of individually addressable caches.
4. The apparatus of claim 3 , wherein:
the LLC is reconfigured with a variable cache size.
5. An electronic device, comprising:
a general purpose graphics processor;
a plurality of compute engines communicatively coupled to a last-level cache (LLC) by an interconnect, the LLC configured with a variable cache line size, wherein the plurality of compute engines execute one or more layers of a deep learning (DL) network using the LLC; and
a controller communicably coupled to the general purpose graphics processor and the plurality of compute engines, the controller to:
assign one or more of the plurality of compute engines as clients of the LLC;
assign a context identifier (ID) to the clients of the LLC;
receive, from the plurality of compute engines, a plurality of cache access requests, each cache access request comprising:
the context ID corresponding to the compute engine which originated the cache access request; and
metadata that indicates a total size of data to be accessed in one or more subsequent data access transactions;
detect a cache line conflict in the LLC between a first cache access request from a first compute engine and a second cache access request from a second compute engine;
responsive to detecting the cache line conflict, implement a context-based cache eviction policy that utilizes the context ID of the first compute engine and the context ID of the second compute engine to determine a cache way in the LLC to evict in order to resolve the cache line conflict, wherein the LLC is dynamically configured with partitions corresponding to each of the clients based on the one or more layers of the DL network that each of the clients execute; and
responsive to a cache miss, select a variable cache line size to allocate and fill from the LLC, wherein the variable cache line size is selected based on the total size of the data indicated in the metadata received with the access request, and based on data utilization of the LLC by the DL network.
6. The electronic device of claim 5 , wherein:
the cache eviction policy is a function of the context identifier.
7. The electronic device of claim 5 wherein:
the LLC is reconfigured dynamically into a plurality of individually addressable caches.
8. The electronic device of claim 7 , wherein:
the LLC is reconfigured with a variable cache size.
9. One or more non-transitory computer-readable medium comprising one or more instructions that when executed on at least one processor configure the at least one processor to perform one or more operations to:
assign one or more of a plurality of compute engines communicatively coupled to a last-level cache (LLC) by an interconnect as clients of the LLC, the LLC configured with a variable cache line size, wherein the plurality of compute engines execute one or more layers of a deep learning (DL) network using the LLC;
assign a context identifier (ID) to the clients of the LLC;
receive, from the plurality of compute engines, a plurality of cache access requests, each cache access request comprising:
the context ID corresponding to the compute engine which originated the cache access request; and
metadata that indicates a total size of data to be accessed in one or more subsequent data access transactions;
detect a cache line conflict in the LLC between a first cache access request from a first compute engine and a second cache access request from a second compute engine;
responsive to detecting the cache line conflict, implement a context-based cache eviction policy that utilizes the context ID of the first compute engine and the context ID of the second compute engine to determine a cache way in the LLC to evict in order to resolve the cache line conflict, wherein the LLC is dynamically configured with partitions corresponding to each of the clients based on the one or more layers of the DL network that each of the clients execute; and
responsive to a cache miss, select a variable cache line size to allocate and fill from the LLC, wherein the variable cache line size is selected based on the total size of the data indicated in the metadata received with the access request, and based on data utilization of the LLC by the DL network.
10. The one or more non-transitory computer-readable medium of claim 9 , wherein:
the cache eviction policy is a function of the context identifier.
11. The one or more non-transitory computer-readable medium of claim 9 , wherein:
the LLC is reconfigured dynamically into a plurality of individually addressable caches.
12. The one or more non-transitory computer-readable medium of claim 11 , wherein:
the LLC is reconfigured with a variable cache size.
13. A method comprising:
assigning one or more of a plurality of compute engines communicatively coupled to a last-level cache (LLC) by an interconnect as clients of the LLC, the LLC configured with a variable cache line size, wherein the plurality of compute engines execute one or more layers of a deep learning (DL) network using the LLC;
assigning a context identifier (ID) to the clients of the LLC;
receiving, from the plurality of compute engines, a plurality of cache access requests, each cache access request comprising:
the context ID corresponding to the compute engine which originated the cache access request; and
metadata that indicates a total size of data to be accessed in one or more subsequent data access transactions;
detecting a cache line conflict in the LLC between a first cache access request from a first compute engine and a second cache access request from a second compute engine;
responsive to detecting the cache line conflict, implementing a context-based cache eviction policy that utilizes the context ID of the first compute engine and the context ID of the second compute engine to determine a cache way in the LLC to evict in order to resolve the cache line conflict, wherein the LLC is dynamically configured with partitions corresponding to each of the clients based on the one or more layers of the DL network that each of the clients execute; and
responsive to a cache miss, selecting a variable cache line size to allocate and fill from the LLC, wherein the variable cache line size is selected based on the total size of the data indicated in the metadata received with the access request, and based on data utilization of the LLC by the DL network.
14. The method of claim 13 , wherein:
the cache eviction policy is a function of the context identifier.
15. The method of claim 13 , wherein:
the LLC is reconfigured dynamically into a plurality of individually addressable caches.
16. The method of claim 15 , wherein:
the LLC is reconfigured with a variable cache size.