QUASI-RECURRENT NEURAL NETWORK
The technology disclosed provides a quasi-recurrent neural network (QRNN) that alternates convolutional layers, which apply in parallel across timesteps, and minimalist recurrent pooling layers that apply in parallel across feature dimensions.
1 . A quasi-recurrent neural network (QRNN) system that increases computational efficiency in neural network sequence modeling, the system comprising:
a convolutional layer that
receives a time series of input vectors,
applies a convolutional filter bank in parallel to time series windows over the input vectors, and
concurrently outputs convolutional vectors for the time series windows,
each of the convolution vectors comprising feature values in an activation vector and in one or more gate vectors, and
the feature values in the gate vectors are parameters that, respectively, apply element-wise by ordinal position to the feature values in the activation vector; and
a pooling layer that
receives the convolutional vectors for the time series windows, and
operates in parallel over feature values of a convolutional vector to concurrently accumulate ordinal position-wise, in a state vector for a current time series window, an ordered set of feature sums in dependence upon
a feature value at a given ordinal position in an activation vector outputted for the current time series window,
one or more feature values at the given ordinal position in one or more gate vectors outputted for the current time series window, and
a feature sum at the given ordinal position in a state vector accumulated for a prior time series window; and
sequentially outputs a state vector for each successive time series window among the time series windows.
2 . The system of claim 1 , wherein dimensionality of the convolutional vectors is augmented relative to dimensionality of the input vectors in dependence upon a number of convolutional filters in the convolutional filter bank.
3 . The system of claim 1 , wherein the input vectors represent elements of an input sequence, and
wherein the pooling layer encodes order and context information of the elements in the state vectors.
4 . The system of claim 3 , wherein the input sequence is a word-level sequence.
5 . The system of claim 3 , wherein the input sequence is a character-level sequence.
6 . The system of claim 1 , wherein the gate vector is a forget gate vector, and
wherein the pooling layer uses a forget gate vector for a current time series window to control accumulation of information from a state vector accumulated for a prior time series window and information from an activation vector for the current time series window.
7 . The system of claim 1 , wherein the gate vector is an input gate vector, and
wherein the pooling layer uses an input gate vector for a current time series window to control accumulation of information from an activation vector for the current time series window.
8 . The system of claim 1 , wherein the gate vector is an output gate vector, and
wherein the pooling layer uses an output gate vector for a current time series window to control accumulation of information from a state vector for the current time series window.
9 . The system of claim 1 , further configured to comprise a plurality of sub-QRNNs arranged in a sequence from lowest to highest, each sub-QRNN comprising at least one convolutional layer and at least one pooling layer.
10 . The system of claim 9 , wherein the sub-QRNNs are configured to:
receive as input a preceding output generated by a preceding sub-QRNN in the sequence;
process the preceding output through the convolutional layer to produce an alternative representation of the preceding output; and
process the alternative representation through the pooling layer to produce an output.
11 . The system of claim 10 , further configured to include skip connections between the sub-QRNNs and between layers in the sub-QRNN,
wherein the skip connections concatenate output of a preceding layer with output of a current layer and provide the concatenation to a following layer as input.
12 . The system of claim 1 , further configured to train convolutional filters in the convolutional filter bank using a sequence task.
13 . The system of claim 12 , wherein the sequence task is language modeling.
14 . The system of claim 12 , wherein the sequence task is sentiment classification.
15 . The system of claim 12 , wherein the sequence task is document classification.
16 . The system of claim 12 , wherein the sequence task is word-level machine translation.
17 . The system of claim 12 , wherein the sequence task is character-level machine translation.
18 . The system of claim 1 , further configured to regularize the QRNN by:
requiring a random subset of feature sums at given ordinal positions in the state vector for the current time series window to replicate respective feature sums at the given ordinal positions in the state vector concurrently accumulated for the prior time series window, including
requiring respective feature values at the given ordinal positions in a forget gate vector for the current time series window to be unity.
19 . A quasi-recurrent neural network (QRNN) system that increases computational efficiency in neural network sequence modeling, the system comprising:
a convolutional layer, running on numerous processing cores, that
receives a time series of input vectors,
applies a convolutional filter bank in parallel to time series windows over the input vectors, and
concurrently outputs convolutional vectors for the time series windows,
each of the convolution vectors comprising feature values in an activation vector and in one or more gate vectors, the gate vectors including a forget gate vector, and
the feature values in the gate vectors are parameters that, respectively, apply element-wise by ordinal position to the feature values in the activation vector; and
a pooling layer, running on numerous processing cores, that
receives the convolutional vectors for the time series windows, and
applies accumulators in parallel to respective feature values of a convolutional vector, to calculate a state vector for each successive timestep among the time series windows; and
at each timestep, for respective ordinal positions in an activation vector and one or more gate vectors of the convolutional vector, an accumulator
begins with a feature sum at a given ordinal position in a state vector from a prior timestep, if any, multiplied by a respective feature value at the given ordinal position in the forget gate vector for a current timestep,
adds an evaluation of a respective feature value at the given ordinal position in the activation vector for the current timestep against one or more respective feature values at the given ordinal position in the gate vectors for the current timestep, and
outputs a state vector for the current timestep that combines results of the accumulators across all of the respective ordinal positions.
20 . The system of claim 19 , wherein the pooling layer uses the forget gate vector for the current timestep to control accumulation of information from the activation vector for the current timestep.
21 . The system of claim 19 , wherein the gate vectors include an input gate vector, and
wherein the pooling layer uses an input gate vector for the current timestep to control accumulation of information from the activation vector for the current timestep.
22 . The system of claim 19 , wherein the gate vectors include an output gate vector, and
wherein the pooling layer uses an output gate vector for the current timestep to control accumulation of information from the state vector for the current timestep.
23 . A method that increases computational efficiency in neural network sequence modeling, the method including:
receiving a time series of input vectors;
applying a convolutional filter bank in parallel to time series windows over the input vectors;
concurrently outputting convolutional vectors for the time series windows,
each of the convolution vectors comprising feature values in an activation vector and in one or more gate vectors, and
the feature values in the gate vectors are parameters that, respectively, apply element-wise by ordinal position to the feature values in the activation vector;
operating in parallel over feature values of a convolutional vector to concurrently accumulate ordinal position-wise, in a state vector for a current time series window, an ordered set of feature sums in dependence upon
a feature value at a given ordinal position in an activation vector outputted for the current time series window,
one or more feature values at the given ordinal position in one or more gate vectors outputted for the current time series window, and
a feature sum at the given ordinal position in a state vector accumulated for a prior time series window; and
sequentially outputting a state vector for each successive time series window among the time series windows.
24 . A quasi-recurrent neural network (QRNN) system that increases computational efficiency in neural network sequence modeling, the system comprising:
a convolutional layer that receives a time series of input vectors and concurrently outputs convolutional vectors for time series windows; and
a pooling layer that receives the convolutional vectors for the time series windows and concurrently accumulates an ordered set of feature values in a state vector for a current time series window, and sequentially outputs a state vector for each successive time series window among the time series windows.
25 . A non-transitory computer readable storage medium impressed with computer program instructions, the instructions, when executed on a processor, implement the system of claim 1 .
26 . A non-transitory computer readable storage medium impressed with computer program instructions, the instructions, when executed on a processor, implement the system of claim 24 .