Fully convolutional color constancy with confidence weighted pooling
Disclosed in some examples, are methods, systems, and machine readable mediums that correct image color casts by utilizing a fully convolutional network (FCN), where the patches in an input image may differ in influence over the color constancy estimation. This influence is formulated as a confidence weight that reflects the value of a patch for inferring the illumination color. The confidence weights are integrated into a novel pooling layer where they are applied to local patch estimates in determining a global color constancy result.
1. A system comprising:
a processor;
a memory storing instructions, which when executed by the processor, causes the processor to perform operations comprising:
receiving a digital image;
applying the digital image as input to a convolutional neural network (CNN), the CNN producing a plurality of local patch estimates of color casts in the digital image and corresponding confidence values for the plurality of local patch estimates, wherein the CNN comprises a layer applying a confidence-weighted pooling operation to the plurality of local patch estimates of the color casts in the digital image and the corresponding confidence values to produce an output color cast;
regressing a function c(R i ) to produce the corresponding confidence values of the plurality of local patch estimates, wherein R i are a set of the plurality of local patch estimates, wherein, in the layer applying the confidence-weighted pooling, the plurality of local patch estimates are weighted by their corresponding confidence values and the weighted local patch estimates are summed to produce the output color cast; and
correcting, based on the output color cast, illumination of the color casts of the digital image to create an output digital image, wherein the CNN is trained using a set of training images, the respective training images labelled with an illumination color cast of the image, and wherein the CNN is trained by minimizing a loss function, the loss function defined as an angular error between an estimated color cast and the labelled illumination color cast.
2. The system of claim 1 , wherein the CNN comprises a convolution layer, a Rectified Linear Unit layer, and a max pooling layer.
3. The system of claim 1 , wherein the CNN is a seven-layer convolutional neural network.
4. The system of claim 1 , wherein the output digital image is input to an object recognition algorithm that recognizes an object in the output digital image.
5. A non-transitory machine-readable medium storing instructions, which when executed by a machine, causes the machine to perform operations comprising:
receiving a digital image;
applying the digital image as input to a convolutional neural network (CNN), the CNN producing a plurality of local patch estimates of color casts in the digital image and corresponding confidence values for the plurality of local patch estimates, wherein the CNN comprises a layer applying a confidence-weighted pooling operation to the plurality of local patch estimates of the color casts in the digital image and the corresponding confidence values to produce an output color cast;
regressing a function c(R i ) to produce the corresponding confidence values of the plurality of local patch estimates, wherein R i are a set of the plurality of local patch estimates, wherein, in the layer applying the confidence-weighted pooling, the plurality of local patch estimates are weighted by their corresponding confidence values and the weighted local patch estimates are summed to produce the output color cast; and
correcting, based on the output color cast, illumination of the color casts of the digital image to create an output digital image, wherein the CNN is trained using a set of training images, the respective training images labelled with an illumination color cast of the image, and wherein the CNN is trained by minimizing a loss function, the loss function defined as an angular error between an estimated color cast and the labelled illumination color cast.
6. The machine-readable medium of claim 5 , wherein the CNN comprises a convolution layer, a Rectified Linear Unit layer, and a max pooling layer.
7. The machine-readable medium of claim 5 , wherein the CNN is a seven-layer convolutional neural network.
8. The machine-readable medium of claim 5 , wherein the output digital image is input to an object recognition algorithm that recognizes an object in the output digital image.
9. A method comprising:
receiving a digital image;
applying the digital image as input to a convolutional neural network (CNN), the CNN producing a plurality of local patch estimates of color casts in the digital image and corresponding confidence values for the plurality of local patch estimates, wherein the CNN comprises a layer applying a confidence-weighted pooling operation to the plurality of local patch estimates of the color casts in the digital image and the corresponding confidence values to produce an output color cast;
regressing a function c(R i ) to produce the corresponding confidence values of the plurality of local patch estimates, wherein R i are a set of the plurality of local patch estimates, wherein, in the layer applying the confidence-weighted pooling, the plurality of local patch estimates are weighted by their corresponding confidence values and the weighted local patch estimates are summed to produce the output color cast; and
correcting, based on the output color cast, illumination of the color casts of the digital image to create an output digital image, wherein the CNN is trained using a set of training images, the respective training images labelled with an illumination color cast of the image, and wherein the CNN is trained by minimizing a loss function, the loss function defined as an angular error between an estimated color cast and the labelled illumination color cast.
10. The method of claim 9 , wherein the CNN comprises a convolution layer, a Rectified Linear Unit layer, and a max pooling layer.
11. The method of claim 9 , wherein the CNN is a seven-layer convolutional neural network.