Class-based training of a machine learning model for image processing
Methods of training a machine learning model for image processing are described. A method of training includes utilising as a learning objective a reduction or minimisation of a combination of both an image loss and a classification loss. A method of training includes utilising unsupervised images pairs generated by applying a selected degradation model to a target image, the selected degradation model being selected based on classification information associated with the target image. Methods for generating unsupervised image pairs and methods for image processing using a trained machine learning model are also described, together with computer systems and computer-readable storage for performing the various methods.
1 . A method of training a machine learning model for image processing, the method including, by a computer processing system implementing a machine learning model:
for an image pair comprising a first image and a second image, wherein the first image is a degraded image, comprising degraded image characteristics relative to the second image, and the second image is a target image for machine learning:
a) applying a current machine learning model to the degraded image to produce a processed image output;
b) determining a loss for training, the loss for training comprising a loss between the processed image output and the target image;
c) updating parameters of the machine learning model based on the loss for training; and
d) performing processes a) to c) for a plurality of other image pairs until an end condition is met, each of the other image pairs being different to the first image pair and each other, and each image pair comprising a said degraded image and a said target image;
wherein:
a plurality of, up to all of, the image pairs are unsupervised image pairs;
a said unsupervised image pair is one in which the degraded image has been generated by a computational process based on the target image of the unsupervised image pair;
the computational process comprises applying a selected degradation model to the target image;
the selected degradation model is one of a plurality of degradation models available for selection;
the selected degradation model for each of the plurality of unsupervised image pairs is selected based on scene classification information associated with the target image of that unsupervised image pair, the scene classification information being or defining a scene class of the target image.
2 . The method of claim 1 , wherein the first image has a plurality of visual parameters with an associated parameter value, affecting how the first image appears relative to the second image.
3 . The method of claim 2 , wherein the plurality of visual parameters include at least one of: (i) brightness, (ii) contrast, (iii) saturation, (iv) vibrance, (v) whites, (vi) blacks, (vii) shadows and (viii) highlights.
4 . The method of claim 2 , wherein the image pairs comprise a first image pair with a first set of the plurality of visual parameters and a second image pair with a second set of the plurality of visual parameters, the first set being different from the second set.
5 . The method of claim 4 , wherein the first set and the second set are mutually exclusive.
6 . The method of claim 4 , wherein the first set and the set include at least one common visual parameter.
7 . The method of claim 2 , wherein the plurality of visual parameters were selected according to a random or quasi-random process.
8 . The method of claim 2 , wherein a first degradation model of the plurality of degradation models is associated with a first range of values for a first visual parameter of the plurality of visual parameters and a second degradation model of the plurality of degradation models is associated with a second range of values, different to the first range of values, for the first of the plurality of visual parameters and wherein the applying either the first or the second degradation model to the target image comprises determining a value for the first visual parameter from the first or the second range of values respectively.
9 . The method of claim 8 , wherein determining a value for the first visual parameter within the first or second range of values comprises a random or quasi-random selection process.
10 . The method of claim 2 , wherein the visual parameters are expressed as differentiable functions.
11 . The method of claim 1 , wherein the classification information associated with at least one of the target images identifies a class of one of: (i) people, (ii) nature, (iii) sunrise and sunset, (iv) animals, (v) city, (vi) food, and (vii) night.
12 . The method of claim 1 , wherein:
a) applying the current machine learning model to the degraded image also produces a first classification output; and
b) the loss for training also comprises a loss between the first classification output and the classification information.
13 . The method of claim 12 , wherein the loss for training is a mathematical combination of the loss between the processed image output and the target image and the loss between the first classification output and the classification information.
14 . A computer processing system including one or more computer processors and computer-readable storage, the computer processing system configured to perform a method comprising:
for an image pair comprising a first image and a second image, wherein the first image is a degraded image, comprising degraded image characteristics relative to the second image, and the second image is a target image for machine learning:
a) applying a current machine learning model to the degraded image to produce a processed image output;
b) determining a loss for training, the loss for training comprising a loss between the processed image output and the target image;
c) updating parameters of the machine learning model based on the loss for training; and
d) performing processes a) to c) for a plurality of other image pairs until an end condition is met, each of the other image pairs being different to the first image pair and each other, and each image pair comprising a said degraded image and a said target image;
wherein:
a plurality of, up to all of, the image pairs are unsupervised image pairs;
a said unsupervised image pair is one in which the degraded image has been generated by a computational process based on the target image of the unsupervised image pair;
the computational process comprises applying a selected degradation model to the target image;
the selected degradation model is one of a plurality of degradation models available for selection;
the selected degradation model for each of the plurality of unsupervised image pairs is selected based on scene classification information associated with the target image of that unsupervised image pair, the scene classification information being or defining a scene class of the target image.
15 . Non-transitory computer-readable storage storing instructions for a computer processing system, wherein the instructions, when executed by the computer processing system cause the computer processing system to perform a method comprising:
for an image pair comprising a first image and a second image, wherein the first image is a degraded image, comprising degraded image characteristics relative to the second image, and the second image is a target image for machine learning:
a) applying a current machine learning model to the degraded image to produce a processed image output;
b) determining a loss for training, the loss for training comprising a loss between the processed image output and the target image;
c) updating parameters of the machine learning model based on the loss for training; and
d) performing processes a) to c) for a plurality of other image pairs until an end condition is met, each of the other image pairs being different to the first image pair and each other, and each image pair comprising a said degraded image and a said target image;
wherein:
a plurality of, up to all of, the image pairs are unsupervised image pairs;
a said unsupervised image pair is one in which the degraded image has been generated by a computational process based on the target image of the unsupervised image pair;
the computational process comprises applying a selected degradation model to the target image;
the selected degradation model is one of a plurality of degradation models available for selection;
the selected degradation model for each of the plurality of unsupervised image pairs is selected based on scene classification information associated with the target image of that unsupervised image pair, the scene classification information being or defining a scene class of the target image.