Techniques for generating training data for machine learning enabled image enhancement
Systems and methods for generating training data using synthesized input images for training a machine learning model for image enhancement in accordance with embodiments of the invention are described. The system may access a target image (e.g., captured by an imaging device), and generate an input image that corresponds to the target image. The input image and target image may then be used (e.g., as part of a training data set) to train the machine learning model. For example, a generated input image corresponding to a target image may represent a version of the target image as if it were captured in low light. The target image may be a target illuminated output to be generated by enhancing the input image. The input image and target image may be used to train a machine learning model to enhance images such as those captured in low light.
1. A method of enhancing a source image using an image enhancement machine learning model comprising:
capturing a source image using a digital imager component of a communication device;
processing the source image using an image enhancement machine learning model on the communication device, wherein the image enhancement machine learning model is trained by:
using at least one processor to perform:
accessing a set of target images, wherein the target images represent a target output of the image enhancement machine learning model;
generating a set of input images, wherein the set of input images correspond to one or more target images in the set of target images and represents content in at least one target image modified to simulate being captured under less ideal lighting conditions than in the at least one target image based on determining one or more potential dark pixel values associated with pixels in the at least one target image;
selecting a first reference pixel, based on a reference image, in response to determining that the first reference pixel is a closest match to the one or more potential dark pixel values; and
training the image enhancement machine learning model using a training dataset,
wherein the training dataset comprises the set of target images and the generated set of input images corresponding to the one or more target images to obtain a trained image enhancement learning model; and
outputting the processed source image.
2. The method of claim 1 , wherein the training the image enhancement machine learning model using the training dataset comprises:
dividing the set of input images into a plurality of input image portions;
dividing the one or more target images into a plurality of target image portions; and
wherein the training dataset comprises the plurality of input image portions and the plurality of target image portions.
3. The method of claim 1 , wherein the processing the source image using the image enhancement machine learning model on the communication device comprises:
dividing the source image into a plurality of source image portions;
obtaining output image portions generated by the image enhancement machine learning model using the plurality of source image portions as inputs; and
combining the output image portions into another processed source image.
4. The method of claim 1 , wherein the generating the set of input images comprises:
for the set of input images:
identifying a first target image pixel value in the at least one target image;
identifying a first set of predetermined pixel values based on the first target image pixel value;
selecting a pixel value, from the first set of predetermined pixel values, mapped to the first reference pixel; and
setting a first pixel in the set of input images to the pixel value selected from the first set of predetermined pixel values.
5. The method of claim 4 , wherein the identifying the first set of predetermined pixel values comprises:
accessing data mapping a first plurality of pixel values to the first reference pixel and a second plurality of pixel values to a second reference pixel value;
selecting the first reference pixel based on a first pixel value at a first pixel in the at least one target image; and
identifying the first plurality of pixel values to be the first set of predetermined pixel values.
6. The method of claim 4 , further comprising generating the first set of predetermined pixel values corresponding to the first target image pixel value.
7. The method of claim 4 , further comprising:
accessing a plurality of images of a reference object;
identifying sets of pixel values from the plurality of images corresponding to respective locations in the reference object;
for the sets of pixel values, determining a respective reference pixel value based on at least some pixel values in the sets of pixel values; and
storing a mapping of reference pixel values to respective pixel values of the sets of pixel values.
8. The method of claim 7 , wherein the identifying the first set of predetermined pixel values comprises:
identifying a first pixel value of the reference pixel values in the mapping based on the first pixel; and
identifying the first set of predetermined pixel values to be a set of pixel values mapped to the first reference pixel in the mapping.
9. The method of claim 4 , wherein the selecting the pixel value from the first set of predetermined pixel values comprises randomly sampling the pixel value from the first set of predetermined pixel values.
10. The method of claim 4 , wherein the first target image pixel value, the first set of predetermined pixel values, the selected pixel value, and the first pixel comprise a Red, Green, Blue (RGB) format, a Bayer pattern format of (R, Gr, Gb, B), a single RGB channel, or a single Bayer pattern channel.
11. The method of claim 1 , further comprising:
identifying a second pixel value at a second pixel in the at least one target image;
identifying a second set of predetermined pixel values based on the second pixel value;
selecting a pixel value from the second set of predetermined pixel values; and
setting a second pixel in the set of input images to the pixel value selected from the second set of predetermined pixel values.
12. The method of claim 1 , further comprising:
accessing an image;
providing the image as input to the trained image enhancement machine learning model to obtain a corresponding output indicating updated pixel values for the image; and
updating the image using the output from the trained image enhancement machine learning model.
13. The method of claim 1 , further comprising:
accessing a plurality of additional target images representing associated target outputs of image enhancement the machine learning model;
generating, for the plurality of additional target images, an associated input image corresponding to the plurality of additional target images; and
training the image enhancement machine learning model using (a) the at least one target image and a generated input image corresponding to the at least one target image and (b) the plurality of additional target images and the generated associated input image, to obtain a trained machine learning model.
14. An apparatus for training a machine learning model for enhancing images, the apparatus comprising:
at least one processor; and
at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one processor, cause the apparatus to:
access a target image representing a target output of the machine learning model;
generate an input image corresponding to the target image, comprising:
identifying a first target image pixel value;
identifying a first set of predetermined pixel values, associated with one or more potential dark pixel values associated with pixels in the target image, based on the first target image pixel value;
selecting a first reference pixel value, based on a reference image, in response to determining that the first reference pixel value is a closest match to the one or more potential dark pixel values:
selecting a pixel value, from the first set of predetermined pixel values, mapped to the first reference pixel value; and
setting a first pixel in the input image to the pixel value selected from the first set of predetermined pixel values; and
train the machine learning model using the target image and the generated input image corresponding to the target image to obtain a trained machine learning model.
15. The apparatus of claim 14 , wherein when the at least one processor further executes the instructions, further causes the apparatus to:
identify a second pixel value at a second pixel in the target image;
identify a second set of predetermined pixel values based on the second pixel value;
select a pixel value from the second set of predetermined pixel values; and
set a second pixel in the input image to the pixel value selected from the second set of predetermined pixel values.
16. The apparatus of claim 14 , wherein the identifying the first set of predetermined pixel values comprises:
accessing data mapping a first plurality of pixel values to the first reference pixel value and a second plurality of pixel values to a second reference pixel value;
selecting the first reference pixel value based on a first pixel value at a first pixel in the target image; and
identifying the first plurality of pixel values to be the first set of predetermined pixel values.
17. The apparatus of claim 14 , wherein when the at least one processor further executes the instructions, further causes the apparatus to perform generating the first set of predetermined pixel values corresponding to the first target image pixel value.
18. The apparatus of claim 14 , wherein when the at least one processor further executes the instructions, further causes the apparatus to:
access a plurality of images of a reference object;
identify sets of pixel values from the plurality of images corresponding to respective locations in the reference object;
for the sets of pixel values, determine a respective reference pixel value based on at least some pixel values in the set of pixel values; and
store a mapping of reference pixel values to respective pixel values of the sets of pixel values.
19. The apparatus of claim 18 , wherein the identifying the first set of predetermined pixel values comprises:
identifying a first pixel value of the reference pixel values in the mapping based on the first pixel; and
identifying the first set of predetermined pixel values to be a set of pixel values mapped to the first reference pixel value in the mapping.
20. The apparatus of claim 14 , wherein the selecting the pixel value from the first set of predetermined pixel values comprises randomly sampling the pixel value from the first set of predetermined pixel values.
21. The apparatus of claim 14 , wherein when the at least one processor further executes the instructions, further cause the apparatus to:
access an image;
provide the image as input to the trained machine learning model to obtain a corresponding output indicating updated pixel values for the image; and
update the image using the output from the trained machine learning model.
22. The apparatus of claim 14 , wherein when the at least one processor further executes the instructions, further cause the apparatus to:
access a plurality of additional target images representing associated target outputs of the machine learning model;
generate, for the plurality of additional target images, an associated input image corresponding to the plurality of additional target images; and
train the machine learning model using (a) the target image and the generated input image corresponding to the target image and (b) the plurality of additional target images and the generated associated input image, to obtain the trained machine learning model.
23. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one processor, cause the at least one processor to perform:
accessing a target image representing a target output of the machine learning model;
generating an input image corresponding to the target image, comprising:
identifying a first target image pixel value associated with the target image;
identifying a first set of predetermined pixel values, associated with one or more potential dark pixel values associated with pixels in the target image, based on the first target image pixel value;
selecting a first reference pixel value, based on a reference image, in response to determining that the first reference pixel value is a closest match to the one or more potential dark pixel values;
selecting a pixel value, from the first set of predetermined pixel values, mapped to the first reference pixel value; and
setting a first pixel in the input image to the pixel value selected from the first set of predetermined pixel values; and
training the machine learning model using the target image and the generated input image corresponding to the target image to obtain a trained machine learning model.