Offloading quantization of directional blocked data formats to near-memory units

1 . A system comprising:

a processing-in-memory (PIM) hardware unit comprising a PIM memory circuit, a at least one PIM arithmetic logic unit, and a counter-based conditional shift circuit;

the PIM memory circuit configured to store data;

a memory controller circuit configured to map the data to the PIM memory circuit; and

the at least one PIM arithmetic logic unit configured to quantize the data from a scalar format to a directional blocked data format.

2 . The system of claim 1 , wherein the data includes unquantized data or previously quantized data.

3 . The system of claim 1 , wherein the PIM memory circuit includes one or more memory banks shared by the at least one PIM arithmetic logic unit.

4 . The system of claim 3 , wherein the memory controller circuit is configured to spread the data over the one or more memory banks.

5 . The system of claim 1 , wherein the scalar format is BF16 and the directional blocked data format is a microscaling format.

6 . The system of claim 1 , wherein the data is configured as an input tensor including a plurality of tiles.

7 . The system of claim 6 , further comprising a plurality of PIM arithmetic logic units including the at least one PIM arithmetic logic unit, and one or more PIM arithmetic logic units of the plurality of PIM arithmetic logic units are configured to process one or more tiles of the plurality of tiles included in the input tensor.

8 . The system of claim 6 , wherein the memory controller circuit is configured to map input elements of a specific tile of the plurality of tiles to a specific single instruction, multiple data lane of the at least one PIM arithmetic logic unit or a specific column of a dynamic random-access memory word of a memory bank of the PIM memory circuit.

9 . The system of claim 8 , wherein the memory controller circuit is further configured to pack multiple tiles of the plurality of tiles into one or more dynamic random-access memory words within one or more memory banks shared by one or more PIM arithmetic logic units of a plurality of PIM arithmetic logic units.

10 . The system of claim 1 , wherein the counter-based conditional shift circuit is configured to lower a number of processing-in-memory compute operations.

11 . The system of claim 1 , wherein the counter-based conditional shift circuit is configured to:

check a shift amount for each lane in the PIM arithmetic logic unit;

based on the shift amount for each lane, generate a mask register;

using the mask register, calculate a minimum shift amount;

update an intra-lane shift command to include the minimum shift amount; and

decrement the shift amount for each lane by the minimum shift amount.

12 . A system comprising:

a counter-based conditional shift circuit configured to:

check a shift amount for each lane in an arithmetic logic unit;

based on the shift amount for each lane, generate a mask register;

using the mask register, calculate a minimum shift amount;

update an intra-lane shift command to include the minimum shift amount; and

decrement the shift amount for each lane by the minimum shift amount.

13 . The system of claim 12 , further comprising the arithmetic logic unit, and wherein the arithmetic logic unit is configured to perform intra-lane shifts when instructed via the intra-lane shift command.

14 . The system of claim 13 , wherein the counter-based conditional shift circuit is further configured to inform a memory controller circuit that all lanes have been shifted and to instruct the memory controller circuit to stop issuing intra-lane shift commands to the arithmetic logic unit.

15 . The system of claim 14 , further comprising the memory controller circuit.

16 . A method comprising:

mapping data to a memory circuit of a processor-in-memory (PIM) hardware unit, wherein the PIM hardware unit includes a PIM memory circuit, different arithmetic logic units, and a counter-based conditional shift circuit;

processing multiple independent tiles in parallel in different arithmetic logic units of the PIM hardware unit;

perform strided mapping to align elements of a specific tile to a specific lane within a specific arithmetic logic unit;

spreading the specific tile over multiple memory banks of the memory circuit that share the specific arithmetic logic unit; and

packing elements from the multiple independent tiles in one or more dynamic random-access memory words.

17 . The method of claim 16 , wherein the data is formatted in a scalar format.

18 . The method of claim 17 , wherein the scalar format is BF16.

19 . The method of claim 17 , further comprising outputting quantized data formatted in a directional blocked data format.

20 . The method of claim 16 , wherein the counter-based conditional shift circuit is configured to perform operations comprising:

checking a shift amount for each lane in the specific arithmetic logic unit;

based on the shift amount for each lane, generating a mask register;

using the mask register, calculating a minimum shift amount;

updating an intra-lane shift command to include the minimum shift amount; and

decrementing the shift amount for each lane by the minimum shift amount.