Methods and systems for parallelizing computations in recurrently connected artificial neural networks
The present invention relates to methods and systems for improving the training and inference speed of recurrently connected artificial neural networks by parallelizing application of one or more network layer's recurrent connection weights across all items in the layer's input sequence. More specifically, the present invention specifies methods and systems for carrying out this parallelization for any recurrent network layer that implements a linear time-invariant (LTI) dynamical system. The method of parallelization involves first computing the impulse response of a recurrent layer, and then convolving this impulse response with all items in the layer's input sequence, thereby producing all of the layer's outputs simultaneously. Systems com-posed of one or more parallelized linear recurrent layers and one or more nonlinear layers are then operated to perform pattern classification, signal processing, data representation, or data generation tasks.
1 . A computer implemented method for improving training speed of a recurrently connected artificial neural network, the method comprising:
defining a linear recurrent layer of the artificial neural network with input of one or more dimensions;
defining at least one other layer that implements a nonlinear layer type;
parallelizing the application of each linear recurrent layer to an input sequence of the linear recurrent layer by:
computing an impulse response of the linear recurrent layer;
convolving the impulse response with all items in the input sequence simultaneously to produce an output sequence of the linear recurrent layer;
computing responses of the at least one other layer using the output sequence;
computing a loss metric from the response of a last network layer of the at least one other layer,
computing parameter gradients across the input sequence in parallel by backpropagating the loss metric through a convolution of the impulse response with the input sequence; and
updating unfixed weights of the artificial neural network using the parameter gradients to minimize the loss metric;
wherein the artificial neural network is operable to perform at least one of a pattern classification, signal processing, data representation, or data generation task.
2 . The method of claim 1 , wherein computations performed by the linear recurrent layer are expressed as a convolution in the time domain.
3 . The method of claim 1 , wherein computations performed by the linear recurrent layer are expressed as an element-wise multiplication in the frequency domain.
4 . The method of claim 1 , wherein the parallelizing is implemented as a matrix-matrix product between the impulse response of the linear recurrent layer and an upper-triangular matrix that includes all intermediate input sequences up to a full input sequence.
5 . The method of claim 1 , wherein an input to the artificial neural network is provided by using previous outputs from another artificial neural network.
6 . The method of claim 1 , wherein training of the artificial neural network is run with parallelization and inference by the artificial neural network is run without parallelization.
7 . The method of claim 1 wherein the linear recurrent layer comprises a fixed linear discrete or continuous transform.
8 . The method of claim 7 wherein the transform is selected from the group consisting of discrete or continuous Legendre Transform, Fourier Transform, Hadamard Transform, Haar Transform, Laplace Transform, Cosine Transform, Fourier-Stieltjes Transform, Gelfand Transform, and Hartley Transform.
9 . The method of claim 8 wherein the linear recurrent layer comprises a windowed version of the transform.
10 . The method of claim 1 wherein the artificial neural network comprises a plurality of linear recurrent layers.
11 . The method of claim 10 wherein the artificial neural network comprises at least one other layer that implements a nonlinear layer type for each linear recurrent layer.
12 . A data processing system, comprising circuitry configured to improve training speed of a recurrently connected artificial neural network by:
defining a linear recurrent layer of the artificial neural network with input of one or more dimensions;
defining at least one other layer that implements a nonlinear layer type;
parallelizing the application of each linear recurrent layer to an input sequence of the linear recurrent layer by:
computing an impulse response of the linear recurrent layer;
convolving the impulse response with all items in the input sequence simultaneously to produce an output sequence of the linear recurrent layer;
computing responses of the at least one other layer using the output sequence;
computing a loss metric from the response of a last network layer of the at least one other layer;
computing parameter gradients across the input sequence in parallel by backpropagating the loss metric through a convolution of the impulse response with the input sequence; and
updating unfixed weights of the artificial neural network using the parameter gradients to minimize the loss metric;
wherein the artificial neural network is operable to perform at least one of a pattern classification, signal processing, data representation, or data generation task.
13 . The data processing system of claim 12 , wherein computations performed by the linear recurrent layer are expressed as a convolution in the time domain or are expressed as an element-wise multiplication in the frequency domain.
14 . The data processing system of claim 12 , wherein the parallelizing is implemented as a matrix-matrix product between the impulse response of the linear recurrent layer and an upper-triangular matrix that includes all intermediate input sequences up to a full input sequence.
15 . The data processing system of claim 12 , wherein an input to the artificial neural network is provided by using previous outputs from another artificial neural network.
16 . The data processing system of claim 12 , wherein the linear recurrent layer comprises a fixed linear discrete or continuous transform.
17 . The data processing system of claim 16 , wherein the transform is selected from the group consisting of discrete or continuous Legendre Transform, Fourier Transform, Hadamard Transform, Haar Transform, Laplace Transform, Cosine Transform, Fourier-Stieltjes Transform, Gelfand Transform, and Hartley Transform.
18 . The data processing system of claim 17 , wherein the linear recurrent layer comprises a windowed version of the transform.
19 . The data processing system of claim 12 wherein the artificial neural network comprises a plurality of linear recurrent layers.
20 . The data processing system of claim 12 wherein the artificial neural network comprises at least one other layer that implements a nonlinear layer type for each linear recurrent layer.