Neural network chip for ear-worn device
A hearing aid may include a neural network chip having tiles arranged in an array, each tile including memory, 16-128 multiplier-accumulator circuits (MACs), and routing circuitry. The memory of each tile may be configured to store a portion of elements of a matrix A comprising weights of a recurrent neural network. Each tile may be configured to receive and store elements of an activation vector X, and all tiles in a column of the array may be configured to receive the same elements of X. The plurality of tiles may be configured to perform a matrix-vector multiplication A*X by performing multiply-and-accumulate sub-operations in parallel among the plurality of tiles. The routing circuitry from the tiles in each respective row of tiles may be configured to combine results of the multiply-and-accumulate sub-operations.
1. An ear-worn device, comprising:
a neural network chip, comprising:
a plurality of tiles arranged in a tile array, each tile comprising memory, a number of multiplier-accumulator circuits (MACs), and routing circuitry;
wherein:
the memory of each tile is configured to store a portion of elements of a matrix A comprising weights of a neural network;
each tile is configured to receive and store elements of an activation vector X, wherein the activation vector X is derived from an input audio signal;
all or a subset of the plurality of tiles are configured to perform a matrix-vector multiplication A*X by performing multiply-and-accumulate sub-operations in parallel among all or the subset of the plurality of tiles;
routing circuitries of tiles in the tile array are configured to combine results of the multiply-and-accumulate sub-operations; and
a tile of the plurality of tiles is configured to share an element of the activation vector X across calculations performed by multiple MACs in the tile.
2. The ear-worn device of claim 1 , wherein the tile of the plurality of tiles is configured, when sharing the element of the activation vector X across the calculations performed by the multiple MACs in the tile, to share the element of the activation vector X across calculations performed by all MACs in the tile.
3. The ear-worn device of claim 1 , wherein the tile of the plurality of tiles is configured, when sharing the element of the activation vector X across the calculations performed by the multiple MACs in the tile, to share the element of the activation vector X across calculations performed by the multiple MACs in the tile on a single clock cycle.
4. The ear-worn device of claim 1 , wherein the tile of the plurality of tiles is configured, when sharing the element of the activation vector X across the calculations performed by the multiple MACs in the tile, to share the element of the activation vector X across calculations performed by all MACs in the tile on a single clock cycle.
5. The ear-worn device of claim 1 , wherein the routing circuitries of the tiles in the tile array are configured, when combining the results of the multiply-and-accumulate sub-operations, to combine the results of the multiply-and-accumulate sub-operations from each respective row of tiles in the tile array.
6. An ear-worn device, comprising:
a neural network chip, comprising:
a plurality of tiles arranged in a tile array, each tile comprising memory, a number of multiplier-accumulator circuits (MACs), and routing circuitry;
wherein:
the memory of each tile is configured to store a portion of elements of a matrix A comprising weights of a neural network;
each tile is configured to receive and store elements of an activation vector X, wherein the activation vector X is derived from an input audio signal;
all or a subset of the plurality of tiles are configured to perform a matrix-vector multiplication A*X by performing multiply-and-accumulate sub-operations in parallel among all or the subset of the plurality of tiles;
routing circuitries of tiles in the tile array are configured to combine results of the multiply-and-accumulate sub-operations; and
a tile of the plurality of tiles is configured to fan out a single element of the activation vector X from the memory to each of multiple MACs in the tile.
7. The ear-worn device of claim 6 , wherein the tile of the plurality of tiles is configured, when fanning out the single element of the activation vector X from the memory to each of the multiple MACs in the tile, to simultaneously fan out the single element of the activation vector X from the memory to each of the multiple MACs in the tile.
8. The ear-worn device of claim 6 , wherein the tile of the plurality of tiles is configured, when fanning out the single element of the activation vector X from the memory to each of the multiple MACs in the tile, to fan out the single element of the activation vector X from the memory to all MACs in the tile.
9. The ear-worn device of claim 6 , wherein the tile of the plurality of tiles is configured, when fanning out the single element of the activation vector X from the memory to each of the multiple MACs in the tile, to simultaneously fan out the single element of the activation vector X from the memory to all MACs in the tile.
10. An ear-worn device, comprising:
a neural network chip, comprising:
a plurality of tiles arranged in a tile array, each tile comprising memory, a number of multiplier-accumulator circuits (MACs), and routing circuitry; and
a vector memory storing an activation vector X derived from an input audio signal;
wherein:
the memory of each tile is configured to store a portion of elements of a matrix A comprising weights of a neural network;
each tile is configured to receive and store elements of the activation vector X;
all or a subset of the plurality of tiles are configured to perform a matrix-vector multiplication A*X by performing multiply-and-accumulate sub-operations in parallel among all or the subset of the plurality of tiles;
routing circuitries of tiles in the tile array are configured to combine results of the multiply-and-accumulate sub-operations; and
groups of tiles in the tile array are coupled to the vector memory only by a single, shared bus.
11. The ear-worn device of claim 10 , wherein each group of the groups of tiles comprises tiles in a single column of the tile array.
12. An ear-worn device, comprising:
a neural network chip, comprising:
a plurality of tiles arranged in a tile array, each tile comprising memory, a number of multiplier-accumulator circuits (MACs), and routing circuitry; and
a vector memory storing an activation vector X derived from an input audio signal;
wherein:
the memory of each tile is configured to store a portion of elements of a matrix A comprising weights of a neural network;
each tile is configured to receive and store elements of the activation vector X;
all or a subset of the plurality of tiles are configured to perform a matrix-vector multiplication A*X by performing multiply-and-accumulate sub-operations in parallel among all or the subset of the plurality of tiles;
routing circuitries of tiles in the tile array are configured to combine results of the multiply-and-accumulate sub-operations; and
at least one tile in the tile array is not configured for transmitting data to at least one adjacent tile in the tile array.
13. The ear-worn device of claim 12 , wherein the at least one tile in the tile array is not configured for transmitting data to another tile in a same column of the tile array.
14. The ear-worn device of claim 12 , wherein the tile array lacks independent connections between adjacent tiles in a column of the tile array.
15. An ear-worn device, comprising:
a neural network chip, comprising:
a plurality of tiles arranged in a tile array, each tile comprising memory, a number of multiplier-accumulator circuits (MACs), and routing circuitry; and
a vector memory storing an activation vector X derived from an input audio signal;
wherein:
the memory of each tile is configured to store a portion of elements of a matrix A comprising weights of a neural network;
each tile is configured to receive and store elements of the activation vector X;
all or a subset of the plurality of tiles are configured to perform a matrix-vector multiplication A*X by performing multiply-and-accumulate sub-operations in parallel among all or the subset of the plurality of tiles;
routing circuitries of tiles in the tile array are configured to combine results of the multiply-and-accumulate sub-operations; and
all memory in the plurality of tiles together comprises no more than approximately 40 Mbits of memory for the weights of the neural network.
16. An ear-worn device, comprising:
a neural network chip, comprising:
a plurality of tiles arranged in a tile array, each tile comprising memory, a number of multiplier-accumulator circuits (MACs), and routing circuitry;
wherein:
the memory of each tile is configured to store a portion of elements of a matrix A comprising weights of a neural network;
each tile is configured to receive and store elements of an activation vector X, wherein the activation vector X is derived from an input audio signal;
all or a subset of the plurality of tiles are configured to perform a matrix-vector multiplication A*X by performing multiply-and-accumulate sub-operations in parallel among all or the subset of the plurality of tiles;
routing circuitries of tiles in the tile array are configured to combine results of the multiply-and-accumulate sub-operations; and
the neural network chip is configured to achieve a power efficiency of at least 4 GOPs/milliwatt when operating at 100 GOPs/second.
17. The ear-worn device of claim 16 , wherein the neural network comprises up to 10M 8-bit weights.
18. The ear-worn device of claim 17 , wherein the 100 GOPs/second are performed on time series data.
19. The ear-worn device of claim 18 , wherein the neural network chip is at 40 degrees Celsius, uses supply voltages between 0.5-1.8V, and is performing operations without idling.
20. The ear-worn device of claim 16 , wherein the neural network chip has an area less than 20 mm 2 .