Activation layers for deep learning networks
Tasks such as object classification from image data can take advantage of a deep learning process using convolutional neural networks. These networks can include a convolutional layer followed by an activation layer, or activation unit, among other potential layers. Improved accuracy can be obtained by using a generalized linear unit (GLU) as an activation unit in such a network, where a GLU is linear for both positive and negative inputs, and is defined by a positive slope, a negative slope, and a bias. These parameters can be learned for each channel or a block of channels, and stacking those types of activation units can further improve accuracy.
1. A computer-implemented method, comprising:
training a convolutional neural network using a set of a training data, the training data including instances of data with determined classifications;
receiving a query from a client device associated with a user;
processing the query using the trained convolutional neural network to determine a classification of a data object represented in the query, the trained convolutional neural network containing at least one convolutional layer and at least one activation layer, the at least one activation layer including a generalized linear unit, the generalized linear unit having a functional form described using a pair of straight lines with three parameters including a first slope in a positive region, a second slope in a negative region, and an offset applied to the first slope and the second slope, the three parameters learnable over at least one input channel;
determining a set of features corresponding to the classification; and
providing, to the client device, information for at least a subset of the set of features.
2. The computer-implemented method of claim 1 , wherein the first slope is associated with positive inputs and the second slope is associated with negative inputs.
3. The computer-implemented method of claim 1 , claim wherein a first activation layer with the three parameters learnable over a single channel is stacked with a second activation layer with the three parameters learnable over a block of input channels.
4. A computer-implemented method, comprising:
training a convolutional neural network using a set of a training data, the training data including instances of data objects with determined classifications;
receiving a query;
processing the query using the trained convolutional neural network to determine a classification of a data segment represented in the query, the trained convolutional neural network containing at least one convolutional layer and at least one activation layer, the at least one activation layer including a generalized linear unit (GLU), the GLU having three parameters including a first slope in a positive region, a second slope in a negative region, and an offset applied to the first slope and the second slope, the three parameters learnable over at least one input channel;
determining a set of data segments corresponding to the classification; and
providing, to a client device, information for at least a subset of the set of data segments.
5. The computer-implemented method of claim 4 , further comprising:
measuring accuracy of the trained convolutional neural network by classifying data objects in a test set.
6. The computer-implemented method of claim 4 , wherein the GLU is linear for positive inputs and negative inputs.
7. The computer-implemented method of claim 4 , wherein the GLU further includes at least one of a pooling layer, a fully connected layer, or a softmax layer.
8. The computer-implemented method of claim 4 , wherein the first slope is associated with positive inputs and the second slope is associated with negative inputs.
9. The computer-implemented method of claim 4 , wherein the query comprises an image, an audio sample, a text input, or a combination thereof.
10. The computer-implemented method of claim 9 , wherein a first activation layer with the three parameters learnable over a single channel is stacked with a second activation layer with the three parameters learnable over a block of input channels.
11. A system, comprising:
at least one processor; and
memory including instructions that, when executed by the at least one processor, cause the system to:
train a convolutional neural network using a set of a training data, the training data including instances of data with determined classifications;
receive query data;
process the query data using the trained convolutional neural network to determine a classification of at least a portion of the query data, the trained convolutional neural network containing at least one convolutional layer and at least one activation layer, the at least one activation layer including a generalized linear unit (GLU), the GLU having three parameters including a first slope in a positive region, a second slope in a negative region, and an offset applied to the first slope and the second slope, the three parameters learnable over at least one input channel;
determine a set of data objects corresponding to the classification; and
provide, in response to a request, information for at least a subset of the set of data objects.
12. The system of claim 11 , wherein the query data is classified using at least one of a computer vision process, an object recognition process, an audio signal analysis process, a speech recognition process, or an item classification process utilizing the trained convolutional neural network.
13. The system of claim 11 , wherein the instructions when executed further cause the system to:
measure accuracy of the trained convolutional neural network after the training by classifying data in a test set.
14. The system of claim 11 , wherein the GLU further includes at least one of a pooling layer, a fully connected layer, or a softmax layer.
15. The system of claim 11 , wherein the first slope is associated with positive inputs and the second slope is associated with negative inputs.
16. The system of claim 11 , wherein a first activation layer with the three parameters learnable over a single channel is stacked with a second activation layer with the three parameters learnable over a block of input channels.
17. The system of claim 11 , wherein the GLU is linear for both positive and negative inputs.