Automatically segmenting and adjusting images
A device automatically segments an image into different regions and automatically adjusts perceived exposure-levels or other characteristics associated with each of the different regions, to produce pictures that exceed expectations for the type of optics and camera equipment being used and in some cases, the pictures even resemble other high-quality photography created using professional equipment and photo editing software. A machine-learned model is trained to automatically segment an image into distinct regions. The model outputs one or more masks that define the distinct regions. The mask(s) are refined using a guided filter or other technique to ensure that edges of the mask(s) conform to edges of objects depicted in the image. By applying the mask(s) to the image, the device can individually adjust respective characteristics of each of the different regions to produce a higher-quality picture of a scene.
1. A computer-implemented method comprising:
receiving, by a processor of a computing device, an original image captured by a camera;
automatically segmenting, by the processor, the original image into multiple regions of pixels by outputting a mask indicating which pixels of the original image or a down-sampled version of the original image are contained within each of the multiple regions;
using a guided filter to add matting to each of the multiple regions to refine the mask to generate a refined mask;
independently adjusting, by the processor, a respective characteristic of each of the multiple regions as indicated by the refined mask;
combining, by the processor, the multiple regions to form a new image after independently adjusting the respective characteristic of each of the multiple regions; and
outputting, by the processor and for display, the new image.
2. The computer-implemented method of claim 1 , wherein automatically segmenting the original image into the multiple regions comprises:
inputting the original image or the downsampled version of the original image into a machine-learned model, wherein the machine-learned model is configured to output the mask indicating which pixels of the original image or the downsampled version of the original image are contained within each of the multiple regions.
3. The computer-implemented method of claim 2 , wherein the mask further indicates a respective degree of confidence associated with each of the pixels of the original image or the downsampled version of the original image that the pixel is contained within each of the multiple regions.
4. The computer-implemented method of any of claim 1 , wherein independently adjusting the respective characteristic of each of the multiple regions comprises independently applying, by the processor, a respective auto-white-balancing to each of the multiple regions.
5. The computer-implemented method of claim 4 , wherein independently applying the respective auto-white-balancing to each of the multiple regions comprises:
determining a respective type classifying each of the multiple regions;
selecting the respective auto-white-balancing for each of the multiple regions based on the respective type; and
applying the respective auto-while-balancing selected for each of the multiple regions.
6. The computer-implemented method of claim 5 , wherein the respective type classifying each of the multiple regions includes a sky region including a pixel representation of a sky or a non-sky region including a pixel representation of one or more objects other than the sky.
7. The computer-implemented method of any of claim 4 , wherein selecting the respective auto-white-balancing for each of the multiple regions based on the respective type comprises selecting a first auto-white-balancing for each of the multiple regions that is determined to be a sky region and selecting a second, different auto-white-balancing for each of the multiple regions that is determined to be a non-sky region.
8. The computer-implemented method of any of claim 1 , wherein independently adjusting the respective characteristic of each of the multiple regions comprises independently adjusting, by the processor and according to a respective type classifying each of the multiple regions, a respective brightness associated with each of the multiple regions.
9. The computer-implemented method of any of claim 1 , wherein independently adjusting the respective characteristic of each of the multiple regions comprises independently removing, by the processor and according to a respective type classifying each of the multiple regions, noise from each of the multiple regions.
10. The computer-implemented method of claim 9 , further comprising:
determining the respective type classifying each of the multiple regions; and
in response to determining the respective type classifying a particular region from the multiple regions is a sky region type, averaging groups of noisy pixels in the particular region from the multiple regions that are a size and frequency that satisfies a threshold.
11. The computer-implemented method of claim 10 , further comprising:
further in response to determining the respective type classifying the particular region from the multiple regions is the sky region type, retaining groups of noisy pixels in the particular region from the multiple regions that are a size and frequency that is less than or greater than the threshold.
12. The computer-implemented method of any of claim 1 , wherein outputting the new image comprises outputting, for display, the new image automatically and in response to receiving an image capture command from an input component of the computing device, the image capture command directing the camera to capture the original image.
13. A computing device comprising at least one processor configured to:
receive an original image captured by a camera;
automatically segment the original image into multiple regions of pixels by outputting a mask indicating which pixels of the original image or a downsampled version of the original image are contained within each of the multiple regions;
use a guided filter to add matting to each of the multiple regions to refine the mask to generate a refined mask;
independently adjust a respective characteristic of each of the multiple regions as indicated by the refined mask;
combine the multiple regions to form a new image after independently adjusting the respective characteristic of each of the multiple regions; and
output, for display, the new image.
14. The computing device of claim 13 , wherein the at least one processor is configured to automatically segment the original image into the multiple regions by:
inputting the original image or the downsampled version of the original image into a machine-learned model, wherein the machine-learned model is configured to output the mask indicating which pixels of the original image or the downsampled version of the original image are contained within each of the multiple regions.
15. The computing device of claim 14 , wherein the mask further indicates a respective degree of confidence associated with each of the pixels of the original image or the downsampled version of the original image that the pixel is contained within each of the multiple regions.
16. The computing device of claim 13 , wherein the at least one processor is configured to independently adjust the respective characteristic of each of the multiple regions by independently applying a respective auto-white-balancing to each of the multiple regions.
17. The computing device of claim 13 , wherein the at least one processor is configured to independently apply the respective auto-white-balancing to each of the multiple regions by:
determining a respective type classifying each of the multiple regions;
selecting the respective auto-white-balancing for each of the multiple regions based on the respective type; and
applying the respective auto-while-balancing selected for each of the multiple regions.
18. A non-transitory computer readable medium comprising program instructions executable by at least one processor to:
receive an original image captured by a camera;
automatically segment the original image into multiple regions of pixels by outputting a mask indicating which pixels of the original image or a downsampled version of the original image are contained within each of the multiple regions;
use a guided filter to add matting to each of the multiple regions to refine the mask to generate a refined mask;
independently adjust a respective characteristic of each of the multiple regions as indicated by the refined mask;
combine the multiple regions to form a new image after independently adjusting the respective characteristic of each of the multiple regions; and
output, for display, the new image.