Partitionable digital hardware system for implementing recurrent neural networks
Recurrent neural networks are efficiently mapped to hardware computation blocks specifically designed for Legendre Memory Unit (LMU) cells, Projected LSTM cells, and Feed Forward cells. Iterative resource allocation algorithms are used to partition recurrent neural networks and time multiplex them onto a spatial distribution of computation blocks, guided by multivariable optimizations for power, performance, and accuracy. Embodiments of the invention provide systems for low power, high performance deployment of recurrent neural networks for battery sensitive applications such as automatic speech recognition (ASR), keyword spotting (KWS), biomedical signal processing, and other applications that involve processing time-series data.
1 . A digital hardware system for implementing a recurrent neural network, the digital hardware system comprising:
at least one Legendre Memory Unit (LMU) cell computation block, each LMU cell computation block receiving input via an input bus, each LMU cell computation block comprising a digital hardware implementation of an encoder cell computation sub block set comprising one or more encoder cell computation sub blocks, a memory computation sub block set comprising one or more memory computation sub blocks, and a hidden computation sub block set comprising one or more hidden computation sub blocks;
each encoder cell computation sub block of the encoder cell computation sub block set connected to the input bus to receive the input, each encoder cell computation sub block of the encoder cell computation sub block set connected via a first bus to receive output from the memory computation sub block set and each encoder cell computation sub block of the encoder cell computation sub block set connected via a second bus to receive output from the hidden computation sub block set;
each memory computation sub block of the memory computation sub block set connected via a third bus to receive output from the encoder cell computation sub block set, and connected via the first bus to receive the output from the memory computation sub block set;
each hidden computation sub block of the hidden computation sub block set connected to the input bus to receive the input, each hidden computation sub block of the hidden computation sub block set connected via the first bus to receive the output from the memory computation sub block set, and each hidden computation sub block of the hidden computation sub block set connected via the second bus to receive the output from the hidden computation sub block set,
an output of the at least one LMU cell computation block connected to a next block of the digital hardware system.
2 . The digital hardware system of claim 1 , wherein:
the output of each memory computation sub block of the memory computation sub block set is transmitted in a multiplexed serial stream on the first bus;
the output of each hidden computation sub block of the hidden computation sub block set is transmitted in a multiplexed serial stream on the second bus; and
the output of each encoder cell computation sub block of the encoder cell computation sub block set is transmitted in a multiplexed serial stream on the third bus.
3 . The digital hardware system of claim 1 , wherein the encoder cell computation sub block set comprises a plurality of encoder cell computation sub blocks.
4 . The digital hardware system of claim 3 , wherein the memory computation sub block set comprises a plurality of memory computation sub blocks.
5 . The digital hardware system of claim 4 , wherein the hidden computation sub block set comprises a plurality of hidden computation sub blocks.
6 . The digital hardware system of claim 5 , wherein the encoder cell computation sub blocks of the encoder cell computation sub block set are configured to complete operations at the same time; the memory computation sub blocks of the memory computation sub block set are configured to complete operations at the same time; and the hidden computation sub blocks of the hidden computation sub block set of are configured to complete operations at the same time.
7 . The digital hardware system of claim 1 , wherein the memory computation sub block set comprises a plurality of memory computation sub blocks.
8 . The digital hardware system of claim 1 , wherein the hidden computation sub block set comprises a plurality of hidden computation sub blocks.
9 . The digital hardware system of claim 1 , wherein each encoder cell computation sub block comprises:
an input multiplexer;
a first portion comprising a first memory block storing weights and a first multiply and accumulate (MAC) unit, the first MAC unit connected to receive input from the input multiplexer and the first memory block;
a second portion comprising a second memory block storing weights and a second MAC unit, the second MAC unit connected to receive input from the input bus and the second memory block;
an adder for summing results of the first and second portions; and
an output memory block for storing a result of the encoder cell computation sub block.
10 . The digital hardware system of claim 9 , wherein each encoder cell computation sub block further comprises an activation block for executing an activation function on output of the adder.
11 . The digital hardware system of claim 1 , wherein each memory computation sub block comprises:
a first portion comprising a first MAC unit, the first MAC unit connected to receive, as input, corresponding first input weights and the output from the memory computation sub block set;
a second portion comprising a second MAC unit, the second MAC unit connected to receive, as input, corresponding second input weights and the output from the encoder cell computation sub block set;
an adder for summing results of the first and second portions; and
an output memory block for storing a result of the memory computation sub block.
12 . The digital hardware system of claim 11 , wherein each memory computation sub block further comprises an activation block for executing an activation function on output of the adder.
13 . The digital hardware system of claim 11 , wherein each memory computation sub block MB-further comprises circuitry for generating the first input weights and the second input weights in real time.
14 . The digital hardware system of claim 11 , wherein each of the first portion and second portion of each memory computation sub block further comprises a corresponding memory block for storing the corresponding first input weights and second input weights.
15 . The digital hardware system of claim 1 , wherein each hidden computation sub block comprises:
an input multiplexer;
a first portion comprising a first memory block storing weights and a first MAC unit, the first MAC unit connected to receive input from the input multiplexer and the first memory block;
a second portion comprising a second memory block storing weights and a second MAC unit, the second MAC unit connected to receive input from the input bus and the second memory block;
an adder for summing results of the first and second portions; and
an output memory block for storing a result of the hidden computation sub block.
16 . The digital hardware system of claim 15 , wherein each hidden computation sub block further comprises an activation block for executing an activation function on output of the adder.
17 . The digital hardware system of claim 1 , wherein the recurrent neural network is configured for processing time series data.