Machine learning segmentation-based tone mapping in high noise and high dynamic range environments or other environments
A method includes obtaining multiple input image frames and generating a high dynamic range (HDR) blended image based on the input image frames, where the HDR blended image has a higher dynamic range than individual ones of the input image frames. The method also includes performing a tone fusion operation on the HDR blended image based on a semantic delta-weight map to generate a fused image. Performing the tone fusion operation includes synthesizing multiple low dynamic range (LDR) images based on the HDR blended image and generating initial weight maps based on the LDR images. Performing the tone fusion operation also includes generating filtered weight maps based on the initial weight maps, the semantic delta-weight map, and a guided filter. Performing the tone fusion operation further includes generating the fused image based on the filtered weight maps and decomposed versions of the LDR images.
1 . A method comprising:
obtaining multiple input image frames;
generating a high dynamic range (HDR) blended image based on the input image frames, the HDR blended image having a higher dynamic range than individual ones of the input image frames; and
performing a tone fusion operation on the HDR blended image based on a semantic delta-weight map to generate a fused image;
wherein performing the tone fusion operation comprises:
synthesizing multiple low dynamic range (LDR) images based on the HDR blended image;
generating initial weight maps based on the LDR images;
generating filtered weight maps based on the initial weight maps, the semantic delta-weight map, and a guided filter; and
generating the fused image based on the filtered weight maps and decomposed versions of the LDR images.
2 . The method of claim 1 , wherein generating the fused image based on the filtered weight maps and the decomposed versions of the LDR images comprises:
performing an image decomposition of each of the LDR images to generate base and detail components of each of the LDR images;
performing a base blending operation based on the filtered weight maps and the base components of the LDR images;
performing a detail blending operation based on the filtered weight maps and the detail components of the LDR images; and
combining results of the base and detail blending operations to generate the fused image.
3 . The method of claim 1 , wherein synthesizing the LDR images based on the HDR blended image comprises:
identifying an image histogram of the HDR blended image;
determining multiple fusion scales based on the image histogram;
multiplying image data of the HDR blended image by the fusion scales and clipping resulting image data to generate clipped image data; and
applying image signal processing (ISP) conversion to the clipped image data in order to generate YUV images.
4 . The method of claim 3 , wherein generating the initial weight maps based on the LDR images comprises:
generating saliency metrics for the YUV images;
generating color saturation metrics for the YUV images using a first lookup table;
generating well-exposedness metrics for the YUV images using a second lookup table; and
combining the saliency, color saturation, and well-exposedness metrics for each of the YUV images and normalizing the combined metrics to generate the initial weight map for each of the YUV images.
5 . The method of claim 1 , wherein generating the filtered weight maps comprises:
generating modified weight maps based on the initial weight maps and the semantic delta-weight map; and
using the guided filter to remove noise from the modified weight maps while preserving edges in the modified weight maps.
6 . The method of claim 1 , further comprising:
generating the semantic delta-weight map;
wherein generating the semantic delta-weight map comprises:
generating a lower-resolution HDR blended image based on the HDR blended image;
generating a lower-resolution LDR blended image based on the lower-resolution HDR blended image;
generating a lower-resolution LDR blended YUV image based on the lower-resolution LDR blended image;
generating a semantic segmentation mask based on the lower-resolution LDR blended YUV image; and
generating the semantic delta-weight map based on the semantic segmentation mask using a mapping.
7 . The method of claim 6 , wherein:
the semantic segmentation mask is generated using a trained machine learning model that processes the lower-resolution LDR blended YUV image; and
the mapping is used to translate different values of different semantic classes in the semantic segmentation mask into corresponding values in the semantic delta-weight map.
8 . An electronic device comprising:
at least one processing device configured to:
obtain multiple input image frames;
generate a high dynamic range (HDR) blended image based on the input image frames, the HDR blended image having a higher dynamic range than individual ones of the input image frames; and
perform a tone fusion operation on the HDR blended image based on a semantic delta-weight map to generate a fused image;
wherein, to perform the tone fusion operation, the at least one processing device is configured to:
synthesize multiple low dynamic range (LDR) images based on the HDR blended image;
generate initial weight maps based on the LDR images;
generate filtered weight maps based on the initial weight maps, the semantic delta-weight map, and a guided filter; and
generate the fused image based on the filtered weight maps and decomposed versions of the LDR images.
9 . The electronic device of claim 8 , wherein, to generate the fused image based on the filtered weight maps and the decomposed versions of the LDR images, the at least one processing device is configured to:
perform an image decomposition of each of the LDR images to generate base and detail components of each of the LDR images;
perform a base blending operation based on the filtered weight maps and the base components of the LDR images;
perform a detail blending operation based on the filtered weight maps and the detail components of the LDR images; and
combine results of the base and detail blending operations to generate the fused image.
10 . The electronic device of claim 8 , wherein, to synthesize the LDR images based on the HDR blended image, the at least one processing device is configured to:
identify an image histogram of the HDR blended image;
determine multiple fusion scales based on the image histogram;
multiply image data of the HDR blended image by the fusion scales and clipping resulting image data to generate clipped image data; and
apply image signal processing (ISP) conversion to the clipped image data in order to generate YUV images.
11 . The electronic device of claim 10 , wherein, to generate the initial weight maps based on the LDR images, the at least one processing device is configured to:
generate saliency metrics for the YUV images;
generate color saturation metrics for the YUV images using a first lookup table;
generate well-exposedness metrics for the YUV images using a second lookup table; and
combine the saliency, color saturation, and well-exposedness metrics for each of the YUV images and normalize the combined metrics to generate the initial weight map for each of the YUV images.
12 . The electronic device of claim 8 , wherein, to generate the filtered weight maps, the at least one processing device is configured to:
generate modified weight maps based on the initial weight maps and the semantic delta-weight map; and
use the guided filter to remove noise from the modified weight maps while preserving edges in the modified weight maps.
13 . The electronic device of claim 8 , wherein:
the at least one processing device is further configured to generate the semantic delta-weight map; and
to generate the semantic delta-weight map, the at least one processing device is configured to:
generate a lower-resolution HDR blended image based on the HDR blended image;
generate a lower-resolution LDR blended image based on the lower-resolution HDR blended image;
generate a lower-resolution LDR blended YUV image based on the lower-resolution LDR blended image;
generate a semantic segmentation mask based on the lower-resolution LDR blended YUV image; and
generate the semantic delta-weight map based on the semantic segmentation mask using a mapping.
14 . The electronic device of claim 13 , wherein:
the at least one processing device is configured to generate the semantic segmentation mask using a trained machine learning model that is configured to process the lower-resolution LDR blended YUV image; and
the at least one processing device is configured to use the mapping to translate different values of different semantic classes in the semantic segmentation mask into corresponding values in the semantic delta-weight map.
15 . A non-transitory machine-readable medium containing instructions that when executed cause at least one processor of an electronic device to:
obtain multiple input image frames;
generate a high dynamic range (HDR) blended image based on the input image frames, the HDR blended image having a higher dynamic range than individual ones of the input image frames; and
perform a tone fusion operation on the HDR blended image based on a semantic delta-weight map to generate a fused image;
wherein the instructions that when executed cause the at least one processor to perform the tone fusion operation comprise instructions that when executed cause the at least one processor to:
synthesize multiple low dynamic range (LDR) images based on the HDR blended image;
generate initial weight maps based on the LDR images;
generate filtered weight maps based on the initial weight maps, the semantic delta-weight map, and a guided filter; and
generate the fused image based on the filtered weight maps and decomposed versions of the LDR images.
16 . The non-transitory machine-readable medium of claim 15 , wherein the instructions that when executed cause the at least one processor to generate the fused image based on the filtered weight maps and the decomposed versions of the LDR images comprise instructions that when executed cause the at least one processor to:
perform an image decomposition of each of the LDR images to generate base and detail components of each of the LDR images;
perform a base blending operation based on the filtered weight maps and the base components of the LDR images;
perform a detail blending operation based on the filtered weight maps and the detail components of the LDR images; and
combine results of the base and detail blending operations to generate the fused image.
17 . The non-transitory machine-readable medium of claim 15 , wherein the instructions that when executed cause the at least one processor to synthesize the LDR images based on the HDR blended image comprise instructions that when executed cause the at least one processor to:
identify an image histogram of the HDR blended image;
determine multiple fusion scales based on the image histogram;
multiply image data of the HDR blended image by the fusion scales and clipping resulting image data to generate clipped image data; and
apply image signal processing (ISP) conversion to the clipped image data in order to generate YUV images.
18 . The non-transitory machine-readable medium of claim 17 , wherein the instructions that when executed cause the at least one processor to generate the initial weight maps based on the LDR images comprise instructions that when executed cause the at least one processor to:
generate saliency metrics for the YUV images;
generate color saturation metrics for the YUV images using a first lookup table;
generate well-exposedness metrics for the YUV images using a second lookup table; and
combine the saliency, color saturation, and well-exposedness metrics for each of the YUV images and normalize the combined metrics to generate the initial weight map for each of the YUV images.
19 . The non-transitory machine-readable medium of claim 15 , wherein:
the non-transitory machine-readable medium further contains instructions that when executed cause the at least one processor to generate the semantic delta-weight map; and
the instructions that when executed cause the at least one processor to generate the semantic delta-weight map comprise instructions that when executed cause the at least one processor to:
generate a lower-resolution HDR blended image based on the HDR blended image;
generate a lower-resolution LDR blended image based on the lower-resolution HDR blended image;
generate a lower-resolution LDR blended YUV image based on the lower-resolution LDR blended image;
generate a semantic segmentation mask based on the lower-resolution LDR blended YUV image; and
generate the semantic delta-weight map based on the semantic segmentation mask using a mapping.
20 . The non-transitory machine-readable medium of claim 19 , wherein:
the instructions when executed cause the at least one processor to generate the semantic segmentation mask using a trained machine learning model that is configured to process the lower-resolution LDR blended YUV image; and
the instructions when executed cause the at least one processor to use the mapping to translate different values of different semantic classes in the semantic segmentation mask into corresponding values in the semantic delta-weight map.