Mixed-precision computations with state compression
A technique for matching the throughput between writing into and reading from a memory can include receiving, in parallel, computational results in a high precision format for storing into the memory at a first frequency, and storing the computational results in the memory. The technique may further include rounding the computational results using round-to-the-nearest-even or stochastic rounding to down-convert the computational results from the high precision format to a low precision format in parallel, and outputting the computational results in the low precision format in parallel from the memory at a second frequency.
1 . An integrated circuit device comprising:
a memory operable to store computational results;
a write interface operable to write the computational results received at a first frequency into the memory;
a read interface operable to read the computational results from the memory and output the computational results at a second frequency, the second frequency being different from the first frequency; and
a down-converter operable to perform down-conversion of the computational results from a high precision format into a low precision format by rounding the computational results,
wherein the down-conversion is performed when writing the computational results into the memory or when reading the computational results from the memory.
2 . The integrated circuit device of claim 1 , wherein the computational results are stored in the high precision format in the memory, and the down-conversion is performed on the computational results when the computational results are read from the memory.
3 . The integrated circuit device of claim 1 , wherein the computational results are stored in the low precision format in the memory, and the down-conversion is performed on the computational results when the computational results are written into the memory.
4 . The integrated circuit device of claim 1 , wherein the integrated circuit device is operable to switch between down-converting the computational results when writing the computational results into the memory and down-converting the computational results when reading the computational results from the memory.
5 . The integrated circuit device of claim 1 , wherein the down-converter is operable to perform round-to-the-nearest-even, round-up, round-down, or round-to-zero on the computational results.
6 . The integrated circuit device of claim 1 , wherein the down-converter is operable to perform stochastic rounding on the computational results.
7 . The integrated circuit device of claim 1 , wherein the down-converter is operable to switch between performing round-to-the-nearest-even and stochastic rounding.
8 . The integrated circuit device of claim 1 , wherein the high precision format is a 32-bit floating-point (FP32) format, and the low precision format is a 16-bit floating-point (FP16) format, a 16-bit brain floating-point (BF16) format, or an 8-bit floating-point (FP8) format.
9 . The integrated circuit device of claim 1 , wherein the first frequency is higher than the second frequency.
10 . The integrated circuit device of claim 1 , wherein the memory includes an adder to accumulate the computational results being written to the memory.
11 . The integrated circuit device of claim 1 , further comprising a processing engine array operable to generate the computational results as partial sums in the high precision format, wherein the processing engine array includes processing engines arranged in rows and columns, wherein each of the processing engines is operable to perform a multiply-and-add operation.
12 . The integrated circuit device of claim 11 , further comprising a compute engine operable to obtain the computational results in the low precision format from the memory, and perform computations on the computational results in the low precision format.
13 . The integrated circuit device of claim 12 , wherein the compute engine includes one or more of an activation engine, a pooling engine, or a vector compute engine.
14 . The integrated circuit device of claim 12 , wherein the compute engine operates at half a frequency of the processing engine array.
15 . The integrated circuit device of claim 1 , wherein the low precision format contains half a number of bits as the high precision format.
16 . A method comprising:
receiving, in parallel, computational results in a high precision format at a write interface for storing into a memory at a first frequency;
storing the computational results in the memory;
rounding the computational results by a down-converter connected to the memory using round-to-the-nearest-even or stochastic rounding to down-convert the computational results from the high precision format to a low precision format in parallel; and
outputting the computational results in the low precision format in parallel from the memory at a second frequency by a read interface, the second frequency being different from the first frequency,
wherein the down-conversion is performed when writing the computational results into the memory or when reading the computational results from the memory.
17 . The method of claim 16 , wherein the computational results are stored in the high precision format in the memory, and the rounding of the computational results is performed after the computational results are read from the memory.
18 . The method of claim 16 , wherein the computational results are stored in the low precision format in the memory, and the rounding of the computational results is performed before writing the computational results into the memory.
19 . The method of claim 16 , wherein the high precision format is a 32-bit floating-point (FP32) format, and the low precision format is a 16-bit floating-point (FP16) format, a 16-bit brain floating-point (BF16) format, or a 8-bit floating-point (FP8) format.
20 . The method of claim 16 , wherein the first frequency is higher than the second frequency.