Method and apparatus for image background removal and replacement
Electronic devices may be provided with image sensors and light sources. The image sensors may include image pixels each having a photosensitive element, first and second storage nodes, and first and second transfer transistors coupled between the photosensitive element and the first and second storage nodes. The first and second transfer transistors may be synchronized with the light source so that charges generated by the photosensitive element of each image pixel when the light source is on are transferred to the first storage node of that pixel and charges generated by the photosensitive element of each image pixel when the light source is off are transferred to the second storage node of that pixel. The light source may be an oscillating light source that is configured to turn on and off multiple times during an image exposure. The generated charges may be used in flash-matting operations.
1. A method of operating an electronic device having a light source and an array of image pixels, wherein each image pixel includes a photosensitive element, first and second storage nodes, and first and second transfer transistors coupled between the photosensitive element and the first and second storage nodes, the method comprising:
with the photosensitive element and the first and second storage node of each image pixel, capturing time-multiplexed flash-image data and no-flash-image data;
generating a flash image and a no-flash image from the time-multiplexed flash-image data and no-flash-image data;
comparing flash-image pixel values of the flash image with no-flash-image pixel values of the no-flash image; and
generating a matte image using the comparison of the flash-image pixel values of the flash image with no-flash-image pixel values of the no-flash image, wherein the matte image includes matte pixel values that indicate whether each pixel in the matte image is a foreground object pixel or a background object pixel, wherein capturing the time-multiplexed flash-image data and no-flash-image data comprises:
turning on the light source at a first time;
turning off the light source at a second time that is after the first time;
turning on the light source at a third time that is after the second time;
turning off the light source at a fourth time that is after the third time;
with the photosensitive element and the first storage node of each image pixel, capturing and storing a first portion of the time-multiplexed flash-image data after the first time and before the second time;
with the photosensitive element and the second storage node of each image pixel, capturing and storing a first portion of the time-multiplexed no-flash-image data after the second time and before the third time;
with the first storage node of each image pixel on which the first portion of the time-multiplexed flash-image data is stored, storing a second portion of the time-multiplexed flash-image data after the third time and before the fourth time;
with the second storage node of each image pixel on which the first portion of the time-multiplexed no-flash-image data is stored, storing a second portion of the time-multiplexed no-flash-image data after the fourth time and before a fifth time that is after the fourth time; and
reading out the first and second portions of the time-multiplexed flash-image data from the first storage node of each image pixel after the fifth time.
2. The method defined in claim 1 wherein the flash image and the no-flash image each include an image of a moving object and wherein the foreground object pixels of the matte image include pixels through which the moving object moved while capturing the time-multiplexed flash-image data and no-flash-image data.
3. The method defined in claim 1 , further comprising:
generating a background-removed image using the time-multiplexed flash-image data and the matte image.
4. The method defined in claim 3 , further comprising:
generating a background-replaced image using the background-removed image and the matte image.
5. The method defined in claim 3 wherein generating the background-removed image using the time multiplexed flash-image data and the matte image comprises:
setting pixel values in the flash image that correspond to background object pixels in the matte image to zero.
6. The method defined in claim 1 wherein capturing the time-multiplexed flash-image data and no-flash-image data further comprises:
after reading out the first and second portions of the time-multiplexed flash-image data from the first storage node of each image pixel, reading out the first and second portions of the time-multiplexed no-flash-image data from the second storage node of each image pixel.
7. The method defined in claim 1 , wherein the second time is before the third time and the third time is before the fourth time.