Reference-based super-resolution for image and video enhancement
Devices, methods, and computer readable media to provide enhanced images in multi-camera systems, e.g., by using images captured by cameras with different optical properties and/or sensors. In one embodiment, the techniques comprise reference-based image super-resolution techniques for producing, with a first neural network employing robust feature aggregation techniques (e.g., techniques able to blend between single-image enhancement and feature aggregation, when appropriate), an enhanced output image that attempts to match the quality characteristics of each of region in a lower quality (e.g., shorter focal length, larger field of view (FOV)) input image with the quality characteristics of the region's determined guidance region from at least a second, i.e., higher quality (e.g., longer focal length, smaller FOV image) input image. The guidance regions from the higher quality image that are determined for each region from the lower quality input image may be determined by performing homographic mapping and/or semantic feature matching techniques.
1. An image processing method, comprising:
obtaining a first image from a first image capture device of an electronic device, wherein the first image has a first field of view (FOV) and a first set of quality characteristics and comprises a first plurality of regions;
obtaining a second image from a second image capture device of the electronic device, wherein the second image has a second FOV and a second set of quality characteristics and comprises a second plurality of regions, wherein the second FOV comprises an overlapping portion with the first FOV and a non-overlapping portion with the first FOV, and wherein the second set of quality characteristics is different than the first set of quality characteristics;
determining a guidance region from the first plurality of regions for each of the second plurality of regions;
obtaining a first neural network, wherein the first neural network is configured to take: the second plurality of regions; and the determined guidance regions for each of the second plurality of regions as inputs; and
producing, with the first neural network, a third image having a third set of quality characteristics and a third FOV, wherein the third image and third set of quality characteristics comprise a result of the first neural network attempting to match the quality characteristics of each of the second plurality of regions to the quality characteristics of its corresponding guidance region from the first plurality of regions.
2. The method of claim 1 , further comprising:
determining a first set of features in the first image;
determining a second set of features in the second image; and
determining, based on the first and second sets of features, a homography mapping for each of the second plurality of regions located in the overlapping portion with a corresponding region from the first plurality of regions.
3. The method of claim 2 , wherein determining a guidance region from the first plurality of regions for each of the second plurality of regions further comprises:
determining a guidance region from the first plurality of regions for each of the second plurality of regions located in the overlapping portion based, at least in part, on the determined homography mappings.
4. The method of claim 3 , further comprising:
determining a semantic mapping for each of the second plurality of regions located in the non-overlapping portion to a corresponding region from the second plurality of regions located in the overlapping portion.
5. The method of claim 4 , wherein determining a guidance region from the first plurality of regions for each of the second plurality of regions further comprises:
determining a guidance region from the first plurality of regions for each of the second plurality of regions located in the non-overlapping portion based, at least in part, on the semantic mappings and the homography mappings.
6. The method of claim 1 , wherein determining a guidance region from the first plurality of regions for each of the second plurality of regions further comprises performing at least one of:
feature-based alignment between pixels in the second plurality of regions located in the overlapping portion and pixels in the first plurality of regions; or
semantic feature matching between pixels in the second plurality of regions located in the overlapping portion and pixels in the second plurality of regions located in the non-overlapping portion.
7. The method of claim 1 , wherein the first image capture device has a higher quality than the second image capture device, wherein the second FOV is larger than the first FOV, and wherein the first FOV is located within the second FOV.
8. The method of claim 1 , wherein the first neural network attempting to match the quality characteristics of each of the second plurality of regions to the quality characteristics of its corresponding guidance region from the first plurality of regions further comprises the first neural network:
performing a feature aggregation technique on pixels from each of the second plurality of regions according to a blending map determined by the first neural network.
9. A non-transitory computer readable medium comprising computer readable instructions executable by one or more processors to:
obtain a first image from a first image capture device of an electronic device, wherein the first image has a first field of view (FOV) and a first set of quality characteristics and comprises a first plurality of regions;
obtain a second image from a second image capture device of the electronic device, wherein the second image has a second FOV and a second set of quality characteristics and comprises a second plurality of regions, wherein the second FOV comprises an overlapping portion with the first FOV and a non-overlapping portion with the first FOV, and wherein the second set of quality characteristics is different than the first set of quality characteristics;
determine a guidance region from the first plurality of regions for each of the second plurality of regions;
obtain a first neural network, wherein the first neural network is configured to take: the second plurality of regions; and the determined guidance regions for each of the second plurality of regions as inputs; and
produce, with the first neural network, a third image having a third set of quality characteristics and a third FOV, wherein the third image and third set of quality characteristics comprise a result of the first neural network attempting to match the quality characteristics of each of the second plurality of regions to the quality characteristics of its corresponding guidance region from the first plurality of regions.
10. The non-transitory computer readable medium of claim 9 , wherein the computer readable instructions are further executable by the one or more processors to:
determine a first set of features in the first image;
determine a second set of features in the second image; and
determine, based on the first and second sets of features, a homography mapping for each of the second plurality of regions located in the overlapping portion with a corresponding region from the first plurality of regions.
11. The non-transitory computer readable medium of claim 10 , wherein the instructions to determine a guidance region from the first plurality of regions for each of the second plurality of regions further comprise instructions to:
determine a guidance region from the first plurality of regions for each of the second plurality of regions located in the overlapping portion based, at least in part, on the determined homography mappings.
12. The non-transitory computer readable medium of claim 11 , wherein the computer readable instructions are further executable by the one or more processors to:
determine a semantic mapping for each of the second plurality of regions located in the non-overlapping portion to a corresponding region from the second plurality of regions located in the overlapping portion.
13. The non-transitory computer readable medium of claim 12 , wherein the instructions to determine a guidance region from the first plurality of regions for each of the second plurality of regions further comprise instructions to:
determine a guidance region from the first plurality of regions for each of the second plurality of regions located in the non-overlapping portion based, at least in part, on the semantic mappings and the homography mappings.
14. The non-transitory computer readable medium of claim 9 , wherein the instructions to determine a guidance region from the first plurality of regions for each of the second plurality of regions further comprise instructions to cause the one or more processors to:
perform feature-based alignment between pixels in the second plurality of regions located in the overlapping portion and pixels in the first plurality of regions; or
perform semantic feature matching between pixels in the second plurality of regions located in the overlapping portion and pixels in the second plurality of regions located in the non-overlapping portion.
15. The non-transitory computer readable medium of claim 9 , wherein the first neural network attempting to match the quality characteristics of each of the second plurality of regions to the quality characteristics of its corresponding guidance region from the first plurality of regions further comprises the first neural network:
performing a feature aggregation technique on pixels from each of the second plurality of regions according to a blending map determined by the first neural network.
16. A device, comprising
a memory;
a first image capture device;
a second image capture device; and
one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions causing the one or more processors to:
obtain a first image from the first image capture device, wherein the first image has a first field of view (FOV) and a first set of quality characteristics and comprises a first plurality of regions;
obtain a second image from the second image capture device, wherein the second image has a second FOV and a second set of quality characteristics and comprises a second plurality of regions, wherein the second FOV comprises an overlapping portion with the first FOV and a non-overlapping portion with the first FOV, and wherein the second set of quality characteristics is different than the first set of quality characteristics;
determine a guidance region from the first plurality of regions for each of the second plurality of regions;
obtain a first neural network, wherein the first neural network is configured to take: the second plurality of regions; and the determined guidance regions for each of the second plurality of regions as inputs; and
produce, with the first neural network, a third image having a third set of quality characteristics and a third FOV, wherein the third image and third set of quality characteristics comprise a result of the first neural network attempting to match the quality characteristics of each of the second plurality of regions to the quality characteristics of its corresponding guidance region from the first plurality of regions.
17. The device of claim 16 , wherein the one or more processors are further configured to execute instructions causing the one or more processors to:
determine a first set of features in the first image;
determine a second set of features in the second image; and
determine, based on the first and second sets of features, a homography mapping for each of the second plurality of regions located in the overlapping portion with a corresponding region from the first plurality of regions.
18. The device of claim 16 , wherein the first image capture device has a higher quality than the second image capture device, wherein the second FOV is larger than the first FOV, and wherein the first FOV is located within the second FOV.
19. The device of claim 16 , wherein the instructions to determine a guidance region from the first plurality of regions for each of the second plurality of regions further comprise instructions to cause the one or more processors to:
perform feature-based alignment between pixels in the second plurality of regions located in the overlapping portion and pixels in the first plurality of regions; or
perform semantic feature matching between pixels in the second plurality of regions located in the overlapping portion and pixels in the second plurality of regions located in the non-overlapping portion.
20. The device of claim 16 , wherein the first neural network attempting to match the quality characteristics of each of the second plurality of regions to the quality characteristics of its corresponding guidance region from the first plurality of regions further comprises the first neural network:
performing a feature aggregation technique on pixels from each of the second plurality of regions according to a blending map determined by the first neural network.