Image classification using batch normalization layers
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing images or features of images using an image classification system that includes a batch normalization layer. One of the systems includes a convolutional neural network configured to receive an input comprising an image or image features of the image and to generate a network output that includes respective scores for each object category in a set of object categories, the score for each object category representing a likelihood that that the image contains an image of an object belonging to the category, and the convolutional neural network comprising: a plurality of neural network layers, the plurality of neural network layers comprising a first convolutional neural network layer and a second neural network layer; and a batch normalization layer between the first convolutional neural network layer and the second neural network layer.
1 . A system comprising:
a user computer; and
a computer system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:
receiving a network input comprising an image or image features of the image from the user computer;
processing the network input using a convolutional neural network configured to receive the network input and to generate a network output that characterizes the image, wherein:
the convolutional neural network includes a first neural network layer and a second neural network layer,
processing the network input using the convolutional neural network comprises processing a first layer input to the first neural network layer in accordance with trained values of a set of parameters of the first neural network layer to generate a first layer output having a plurality of components,
the trained values of the set of parameters of the first neural network layer are a result of training the neural network using a plurality of batches of training data,
each batch of training data comprises a respective plurality of training examples, and
the training of the neural network to determine the trained values of the set of parameters of the first neural network layer comprises, for each of the plurality of batches:
receiving a respective first layer output generated by the first neural network layer for each of the plurality of training examples in the batch;
computing a plurality of normalization statistics for the batch from the first layer outputs, comprising:
determining, for each of a plurality of subsets of the plurality of the components of the first layer outputs, a mean of the components of the first layer outputs for each of the plurality of training examples in the batch that are in the respective subset, and
determining, for each of the plurality of subsets of the plurality of the components of the first layer outputs, a standard deviation of the components of the first layer outputs for each of the plurality of training examples in the batch that are in the respective subset;
generating a respective batch normalization layer output for each training example in the batch, comprising:
for each first layer output and for each of the plurality of subsets, normalizing the components of the first layer output that are in the respective subset using the mean for the respective subset and the standard deviation for the respective subset; and
generating a respective batch normalization layer output for each of the training examples from the normalized layer outputs; and
providing the respective batch normalization layer outputs as inputs to the second neural network layer; and
providing the network output as output of the computer system.
2 . The system of claim 1 , wherein training the neural network to determine the trained values of the set of parameters of the first neural network layer further comprises, for each of the plurality of batches:
generating a respective network output for each training example, comprising processing the respective batch normalization layer outputs using the second neural network layer; and
updating the set of parameters of the first neural network layer using the respective network outputs using a backpropagation technique.
3 . The system of claim 2 , wherein updating the set of parameters of the first neural network layer using the respective network outputs using a backpropagation technique comprises:
backpropagating through the normalization statistics.
4 . The system of claim 1 , wherein the plurality of the components of the first layer output are indexed by dimension, and wherein computing a plurality of normalization statistics for the first layer outputs comprises:
computing, for each of the dimensions, a mean of the components of the first layer outputs in the dimension; and
computing, for each of the dimensions, a standard deviation of the components of the first layer outputs in the dimension.
5 . The system of claim 4 , wherein normalizing each of the plurality of the components of each first layer output comprises:
normalizing the component using the computed mean and computed standard deviation for the dimension corresponding to the component.
6 . The system of claim 4 , wherein generating the respective batch normalization layer output for each of the training examples from the normalized layer outputs comprises:
transforming, for each dimension, the component of the normalized layer output for the training example in the dimension in accordance with current values of a set of parameters for the dimension.
7 . The system of claim 1 , wherein the first neural network layer is a convolutional layer, wherein the plurality of the components of the first layer output are indexed by feature index and spatial location index, and wherein computing a plurality of normalization statistics for the first layer outputs comprises, for each of the feature indices:
computing a mean of the components of the first layer outputs that correspond to the feature index; and
computing a variance of the components of the first layer outputs that correspond to the feature index.
8 . The system of claim 7 , wherein normalizing each of the plurality of the components of each layer output comprises:
normalizing the component using the mean and the variance for the feature index corresponding to the component.
9 . The system of claim 7 , wherein generating the respective batch normalization layer output for each of the training examples from the normalized layer outputs comprises:
transforming each of the plurality of the components of the normalized layer output in accordance with current values of a set of parameters for the feature index corresponding to the component.
10 . One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
receiving a network input comprising an image or image features of the image from the user computer;
processing the network input using a convolutional neural network configured to receive the network input and to generate a network output that characterizes the image, wherein:
the convolutional neural network includes a first neural network layer and a second neural network layer,
processing the network input using the convolutional neural network comprises processing a first layer input to the first neural network layer in accordance with trained values of a set of parameters of the first neural network layer to generate a first layer output having a plurality of components,
the trained values of the set of parameters of the first neural network layer are a result of training the neural network using a plurality of batches of training data,
each batch of training data comprises a respective plurality of training examples, and
the training of the neural network to determine the trained values of the set of parameters of the first neural network layer comprises, for each of the plurality of batches:
receiving a respective first layer output generated by the first neural network layer for each of the plurality of training examples in the batch;
computing a plurality of normalization statistics for the batch from the first layer outputs, comprising:
determining, for each of a plurality of subsets of the plurality of the components of the first layer outputs, a mean of the components of the first layer outputs for each of the plurality of training examples in the batch that are in the respective subset, and
determining, for each of the plurality of subsets of the plurality of the components of the first layer outputs, a standard deviation of the components of the first layer outputs for each of the plurality of training examples in the batch that are in the respective subset;
generating a respective batch normalization layer output for each training example in the batch, comprising:
for each first layer output and for each of the plurality of subsets, normalizing the components of the first layer output that are in the respective subset using the mean for the respective subset and the standard deviation for the respective subset; and
generating a respective batch normalization layer output for each of the training examples from the normalized layer outputs; and
providing the respective batch normalization layer outputs as inputs to the second neural network layer; and
providing the network output as output of the one or more computers.
11 . The non-transitory computer-readable storage media of claim 10 , wherein training the neural network to determine the trained values of the set of parameters of the first neural network layer further comprises, for each of the plurality of batches:
generating a respective network output for each training example, comprising processing the respective batch normalization layer outputs using the second neural network layer; and
updating the set of parameters of the first neural network layer using the respective network outputs using a backpropagation technique.
12 . The non-transitory computer-readable storage media of claim 11 , wherein updating the set of parameters of the first neural network layer using the respective network outputs using a backpropagation technique comprises:
backpropagating through the normalization statistics.
13 . The non-transitory computer-readable storage media of claim 11 , wherein the plurality of the components of the first layer output are indexed by dimension, and wherein computing a plurality of normalization statistics for the first layer outputs comprises:
computing, for each of the dimensions, a mean of the components of the first layer outputs in the dimension; and
computing, for each of the dimensions, a standard deviation of the components of the first layer outputs in the dimension.
14 . The non-transitory computer-readable storage media of claim 13 , wherein normalizing each of the plurality of the components of each first layer output comprises:
normalizing the component using the computed mean and computed standard deviation for the dimension corresponding to the component.
15 . The non-transitory computer-readable storage media of claim 11 , wherein the first neural network layer is a convolutional layer, wherein the plurality of the components of the first layer output are indexed by feature index and spatial location index, and wherein computing a plurality of normalization statistics for the first layer outputs comprises, for each of the feature indices:
computing a mean of the components of the first layer outputs that correspond to the feature index; and
computing a variance of the components of the first layer outputs that correspond to the feature index.
16 . The non-transitory computer-readable storage media of claim 15 , wherein normalizing each of the plurality of the components of each layer output comprises:
normalizing the component using the mean and the variance for the feature index corresponding to the component.
17 . The non-transitory computer-readable storage media of claim 15 , wherein generating the respective batch normalization layer output for each of the training examples from the normalized layer outputs comprises:
transforming each of the plurality of the components of the normalized layer output in accordance with current values of a set of parameters for the feature index corresponding to the component.
18 . A method performed by one or more computers, the method comprising:
receiving a network input comprising an image or image features of the image from the user computer;
processing the network input using a convolutional neural network configured to receive the network input and to generate a network output that characterizes the image, wherein:
the convolutional neural network includes a first neural network layer and a second neural network layer,
processing the network input using the convolutional neural network comprises processing a first layer input to the first neural network layer in accordance with trained values of a set of parameters of the first neural network layer to generate a first layer output having a plurality of components,
the trained values of the set of parameters of the first neural network layer are a result of training the neural network using a plurality of batches of training data,
each batch of training data comprises a respective plurality of training examples, and
the training of the neural network to determine the trained values of the set of parameters of the first neural network layer comprises, for each of the plurality of batches:
receiving a respective first layer output generated by the first neural network layer for each of the plurality of training examples in the batch;
computing a plurality of normalization statistics for the batch from the first layer outputs, comprising:
determining, for each of a plurality of subsets of the plurality of the components of the first layer outputs, a mean of the components of the first layer outputs for each of the plurality of training examples in the batch that are in the respective subset, and
determining, for each of the plurality of subsets of the plurality of the components of the first layer outputs, a standard deviation of the components of the first layer outputs for each of the plurality of training examples in the batch that are in the respective subset;
generating a respective batch normalization layer output for each training example in the batch, comprising:
for each first layer output and for each of the plurality of subsets, normalizing the components of the first layer output that are in the respective subset using the mean for the respective subset and the standard deviation for the respective subset; and
generating a respective batch normalization layer output for each of the training examples from the normalized layer outputs; and
providing the respective batch normalization layer outputs as inputs to the second neural network layer; and
providing the network output as output of the one or more computers.
19 . The method of claim 18 , wherein training the neural network to determine the trained values of the set of parameters of the first neural network layer further comprises, for each of the plurality of batches:
generating a respective network output for each training example, comprising processing the respective batch normalization layer outputs using the second neural network layer; and
updating the set of parameters of the first neural network layer using the respective network outputs using a backpropagation technique.
20 . The method of claim 19 , wherein updating the set of parameters of the first neural network layer using the respective network outputs using a backpropagation technique comprises:
backpropagating through the normalization statistics.