Techniques to utilize near memory compute circuitry for memory-bound workloads

1 . A system comprising:

an input/output (I/O) switch arranged to couple with a plurality of memory devices, a first portion of the plurality of memory devices configured in a first memory pool and a second portion of the plurality of memory devices configured in a second memory pool, the first and second memory pools accessible to one or more host central processing units (CPUs) through the I/O switch;

a first circuitry and a first memory controller resident on the I/O switch, the first circuitry to access the first memory pool using the first memory controller;

a second circuitry and a second memory controller resident on the I/O switch, the second circuitry to access the second memory pool using the second memory controller, wherein the second circuitry is to:

receive a request from a first host CPU to obtain data from the second memory pool and process the data;

obtain the data from the second memory pool based on memory address information included in the request from the first host CPU;

process the data to generate a result for the first host CPU; and

cause the result for the first host CPU to be stored to a memory accessible to the first host CPU.

2 . The system of claim 1 , wherein the first circuitry is to:

receive a request from a second host CPU to obtain data from the first memory pool and process the data;

obtain the data from the first memory pool based on memory address information included in the request from the second host CPU;

process the data to generate a result for the second host CPU; and

cause the result for the second host CPU to be stored to a memory accessible to the second host CPU.

3 . The system of claim 2 , wherein the separate requests from the first and second host CPUs are to accelerate execution of respective workloads of the first and second host CPUs, the respective workloads to include a hierarchical navigable small worlds (HNSW) workload, a deep learning recommendation model (DLRM) workload, a graph neural networks (GNN) workload, or a low-batch matrix-matrix multiplication workload.

4 . The system of claim 2 , comprising the I/O switch configured to operate as a Compute Express Link (CXL) switch, wherein the second circuitry to cause the result for the first host CPU to be stored to the memory accessible to the first host CPU includes the second circuitry to cause the results for the first host CPU to be stored to a memory that is local to the second circuitry and separate from memory included in the second memory pool, wherein the second circuitry is further to:

indicate to the second host CPU via use of CXL.cache protocols that the results for the first host CPU have been stored in the memory for the first host CPU.

5 . The system of claim 1 , wherein to obtain the data from the second memory pool further includes the second circuitry to pre-fetch at least a portion of the data to a cache memory that is local to the second circuitry and separate from memory included in the second memory pool, the at least a portion of the data pre-fetched to the cache memory based on a previous request to obtain and process data received from the first host CPU.

6 . The system of claim 1 , comprising the first circuitry, the second circuitry and the I/O switch are resident on a device that also includes the plurality of memory devices, wherein the plurality of memory devices include a first portion of memory devices arranged to include volatile types of memory and a second portion of memory devices arranged to include non-volatile types of memory.

7 . The system of claim 1 , comprising the I/O switch configured to operate as a Compute Express Link (CXL) switch, wherein the second circuitry to cause the result for the first host CPU to be stored to the memory accessible to the first host CPU includes the second circuitry to cause the results for the first host CPU to be stored to a memory included in the second memory pool, wherein the second circuitry is further to:

indicate to the first host CPU via use of CXL.mem protocols that the results for the first host CPU have been stored in the memory included in the second memory pool.

8 . A method comprising:

receiving a request at first circuitry resident on an input/output (I/O) switch that is arranged to couple with a plurality of memory devices configured in a first memory pool and a second portion of the plurality of memory devices configured in a second memory pool, the first and second memory pools accessible to one or more host central processing units (CPUs) through the I/O switch, wherein the I/O switch also has a first memory controller to enable the first circuitry to access the first memory pool and also includes a second circuitry and a second memory controller resident on the I/O switch, the second memory controller to enable the second circuitry to access the second memory pool, and wherein the request is received from a first host CPU for obtaining data from the first memory pool and for processing the obtained data;

obtaining, by the first circuitry, the data from the first memory pool based on memory address information included in the request from the first host CPU;

processing, by the first circuitry, the data to generate a result for the first host CPU; and

cause, by the first circuitry, the result for the first host CPU to be stored to a memory accessible to the first host CPU.

9 . The method of claim 8 , further comprising:

receiving, at the second circuitry, a request from a second host CPU for obtaining data from the second memory pool and processing the data obtained from the second memory pool;

obtaining, by the first circuitry, the data from the second memory pool based on memory address information included in the request from the second host CPU;

processing, by the first circuitry, the data to generate a result for the second host CPU; and

causing, by the first circuitry, the result for the second host CPU to be stored to a memory accessible to the second host CPU.

10 . The method of claim 9 , wherein the separate requests from the first and second host CPUs are to accelerate execution of respective workloads of the first and second host CPUs, the respective workloads to include a hierarchical navigable small worlds (HNSW) workload, a deep learning recommendation model (DLRM) workload, a graph neural networks (GNN) workload, or a low-batch matrix-matrix multiplication workload.

11 . The method of claim 9 , comprising the I/O switch configured to operate as a Compute Express Link (CXL) switch, wherein the first circuitry causing the result for the first host CPU to be stored to the memory accessible to the first host CPU includes the first circuitry causing the results for the first host CPU to be stored to a memory that is local to the first circuitry and separate from memory included in the first memory pool, wherein the method further comprises:

indicating to the second host CPU, by the first circuitry, via use of CXL.cache protocols that the results for the first host CPU have been stored in the memory for the first host CPU.

12 . The method of claim 8 , wherein obtaining the data from the first memory pool further includes the first circuitry to pre-fetch at least a portion of the data to a cache memory that is local to the first circuitry and separate from memory included in the first memory pool, the at least a portion of the data pre-fetched to the cache memory based on a previous request to obtain and process data received from the first host CPU.

13 . The method of claim 8 , comprising the first circuitry, the second circuitry and the I/O switch are resident on a device that also includes the plurality of memory devices, wherein the plurality of memory devices include a first portion of memory devices arranged to include volatile types of memory and a second portion of memory devices arranged to include non-volatile types of memory.

14 . The method of claim 8 , comprising the I/O switch configured to operate as a Compute Express Link (CXL) switch, wherein the first circuitry causing the result for the first host CPU to be stored to the memory accessible to the first host CPU includes the first circuitry causing the results for the first host CPU to be stored to a memory included in the first memory pool, wherein the method further comprises:

indicating to the first host CPU, by the first circuitry, via use of CXL.mem protocols that the results for the first host CPU have been stored in the memory included in the first memory pool.