Machine learning model training framework for multi-frame image processing
A method for training data generation includes obtaining a first set of image frames of a scene and a second set of image frames of the scene using multiple exposure settings. The method also includes generating an alignment map, a blending map, and an input image using the first set of image frames. The method further includes generating a ground truth image using the alignment map, the blending map, and the second set of image frames. In addition, the method includes using the ground truth image and the input image as an image pair in a training dataset when training a machine learning model to reduce image distortion and noise.
1 . A method for training data generation, the method comprising:
obtaining a first set of image frames of a scene and a second set of image frames of the scene using multiple exposure settings, the first set of image frames having higher noise than the second set of image frames;
generating an alignment map and a blending map using the first set of image frames;
generating an input image using the alignment map, the blending map and the first set of image frames;
generating a ground truth image using the alignment map, the blending map, and the second set of image frames; and
using the ground truth image and the input image as an image pair in a training dataset when training a machine learning model to reduce image distortion and noise.
2 . The method of claim 1 , wherein generating the alignment map and the blending map comprises:
performing a multi-frame processing operation to generate the alignment map and the blending map using the first set of image frames.
3 . The method of claim 1 , wherein generating the alignment map and the blending map comprises:
aligning the first set of image frames to generate the alignment map;
applying the alignment map to the first set of image frames to generate aligned image frames;
identifying motion between the aligned image frames; and
generating the blending map based on the identified motion between the aligned image frames.
4 . The method of claim 3 , wherein generating the input image comprises:
blending the aligned image frames based on the blending map to generate the input image.
5 . The method of claim 1 , wherein generating the ground truth image comprises:
applying the alignment map to the second set of image frames to generate aligned image frames; and
blending the aligned image frames based on the blending map to generate the ground truth image.
6 . The method of claim 1 , wherein:
the first set of image frames is captured using one or more ISO values; and
the second set of image frames is captured using at least one ISO value that is lower than the one or more ISO values.
7 . The method of claim 1 , wherein:
the alignment map provides a common alignment for the ground truth image and the input image; and
the blending map provides a common brightness level for the ground truth image and the input image.
8 . An electronic device comprising:
at least one processing device configured to:
obtain a first set of image frames of a scene and a second set of image frames of the scene using multiple exposure settings, the first set of image frames having higher noise than the second set of image frames;
generate an alignment map and a blending map using the first set of image frames;
generate an input image using the alignment map, the blending map, and the first set of image frames;
generate a ground truth image using the alignment map, the blending map, and the second set of image frames; and
use the ground truth image and the input image as an image pair in a training dataset when training a machine learning model to reduce image distortion and noise.
9 . The electronic device of claim 8 , wherein the at least one processing device is configured to perform a multi-frame processing operation to generate the alignment map and the blending map using the first set of image frames.
10 . The electronic device of claim 8 , wherein, to generate the alignment map and the blending map, the at least one processing device is configured to:
align the first set of image frames to generate the alignment map;
apply the alignment map to the first set of image frames to generate aligned image frames;
identify motion between the aligned image frames; and
generate the blending map based on the identified motion between the aligned image frames.
11 . The electronic device of claim 10 , wherein, to generate the input image, the at least one processing device is configured to blend the aligned image frames based on the blending map to generate the input image.
12 . The electronic device of claim 8 , wherein, to generate the ground truth image, the at least one processing device is configured to:
apply the alignment map to the second set of image frames to generate aligned image frames; and
blend the aligned image frames based on the blending map to generate the ground truth image.
13 . The electronic device of claim 8 , wherein:
the first set of image frames is captured using one or more ISO values; and
the second set of image frames is captured using at least one ISO value that is lower than the one or more ISO values.
14 . The electronic device of claim 8 , wherein:
the alignment map provides a common alignment for the ground truth image and the input image; and
the blending map provides a common brightness level for the ground truth image and the input image.
15 . A non-transitory machine-readable medium containing instructions that when executed cause at least one processor of an electronic device to:
obtain a first set of image frames of a scene and a second set of image frames of the scene using multiple exposure settings, the first set of image frames having higher noise than the second set of image frames;
generate an alignment map and a blending map using the first set of image frames;
generate an input image using the alignment map, the blending map, and the first set of image frames;
generate a ground truth image using the alignment map, the blending map, and the second set of image frames; and
use the ground truth image and the input image as an image pair in a training dataset when training a machine learning model to reduce image distortion and noise.
16 . The non-transitory machine-readable medium of claim 15 , wherein the instructions that when executed cause the at least one processor to generate the alignment map and the blending map comprise:
instructions that when executed cause at least one processor to perform a multi-frame processing operation to generate the alignment map and the blending map using the first set of image frames.
17 . The non-transitory machine-readable medium of claim 15 , wherein the instructions that when executed cause the at least one processor to generate the alignment map and the blending map comprise:
instructions that when executed cause at least one processor to:
align the first set of image frames to generate the alignment map;
apply the alignment map to the first set of image frames to generate aligned image frames;
identify motion between the aligned image frames; and
generate the blending map based on the identified motion between the aligned image frames.
18 . The non-transitory machine-readable medium of claim 17 , wherein the instructions that when executed cause the at least one processor to generate the input image comprise:
instructions that when executed cause at least one processor to blend the aligned image frames based on the blending map to generate the input image.
19 . The non-transitory machine-readable medium of claim 15 , wherein the instructions that when executed cause the at least one processor to generate the ground truth image comprise:
instructions that when executed cause at least one processor to:
apply the alignment map to the second set of image frames to generate aligned image frames; and
blend the aligned image frames based on the blending map to generate the ground truth image.
20 . An electronic device comprising:
at least one imaging sensor; and
at least one processing device configured to:
obtain, using the at least one imaging sensor, multiple image frames of a scene during a multi-frame capture operation;
generate a blended image using the multiple image frames; and
reduce image distortion and noise in the blended image using a trained machine learning model;
wherein the trained machine learning model is trained using multiple image pairs, each of the image pairs including (i) an input image generated from a first set of image frames using an alignment map and a blending map, the alignment map and the blending map generated from the first set of image frames, and (ii) a ground truth image generated from a second set of image frames using the alignment map and the blending map, the first set of image frames having higher noise than the second set of image frames.
21 . The method of claim 1 , further comprising:
generating a denoised image using the machine learning model based on a single frame output from a multi-frame processing operation, wherein the denoised image comprises 12-bit or 16-bit RGB data.