Apparatus and method with image processing for sparse dual-pixel image data

1 . An apparatus comprising:

a processor configured to:

generate a first interpolated image by performing interpolation on a first image and generate a second interpolated image by performing interpolation on a second image, wherein the first image and the second image comprise pixel values obtained by dual-pixels of an image sensor; and

generate a disparity map based on the first interpolated image and the second interpolated image,

wherein the processor is further configured to perform interpolation such that a ratio of a number of interpolation pixels in a horizontal direction to a number of interpolation pixels in a vertical direction is an integer multiple of a ratio of a number of pixels in a horizontal direction of the image sensor to a number of pixels in a vertical direction of the image sensor, where the integer multiple is at least 2.

2 . The apparatus of claim 1 , wherein the first image is a left image, and the second image is a right image.

3 . The apparatus of claim 1 , wherein the processor is further configured to generate the first interpolated image based on a position of a first dual-pixel of the dual-pixels, a first pixel value obtained by the first dual-pixel, a position of a second dual-pixel of the dual-pixels, and a second pixel value obtained by the second dual-pixel.

4 . The apparatus of claim 3 , wherein the processor is configured to generate the second interpolated image based on the position of the first dual-pixel, a third pixel value obtained by the first dual-pixel, the position of the second dual-pixel, and a fourth pixel value obtained by the second dual-pixel.

5 . The apparatus of claim 3 , wherein each of the dual-pixels respectively comprises a first photodiode, a second photodiode, and a single lens configured to refract light to the first photodiode and the second photodiode.

6 . The apparatus of claim 5 , wherein the first pixel value is obtained by the first photodiode of the first dual-pixel, the second pixel value is obtained by the first photodiode of the second dual-pixel, the third pixel value is obtained by the second photodiode of the first dual-pixel, and the fourth pixel value is obtained by the second photodiode of the second dual-pixel.

7 . The apparatus of claim 5 , wherein the apparatus comprises the image sensor.

8 . The apparatus of claim 4 , wherein the processor is further configured to generate an interpolation pixel at an intermediate position between the position of the first dual-pixel and the position of the second dual-pixel.

9 . The apparatus of claim 8 , wherein the processor is further configured to determine an average of the first pixel value and the second pixel value and to set a pixel value of the interpolation pixel to the average.

10 . The apparatus of claim 4 , wherein the processor comprises:

an adder configured to perform addition of the first pixel value and the second pixel value; and

a shifter configured to perform a division operation by performing a bit-shift operation on a result of the addition that is outputted from the adder.

11 . The apparatus of claim 1 , wherein the apparatus comprises a smartphone.

12 . A processor-implemented method comprising:

accessing a first image and a second image, the first image and the second image generated by an image sensor comprising pixels that include dual-pixels and that include pixels that are not dual-pixels, wherein each dual-pixel respectively comprises a first photodiode, a second photodiode, and a single lens, wherein the first image comprises pixel values obtained from the first photodiodes and wherein the second image comprises pixel values obtained from the second photodiodes;

generating a first interpolated image by generating first interpolated pixels based on interpolation of the pixel values in the first image, and generating a second interpolated image by generating second interpolated pixels based on interpolation of the pixel values in the second image; and

generating a disparity map based on the first interpolated image and the second interpolated image,

wherein the processor is further configured to perform interpolation such that a ratio of a number of interpolation pixels in a horizontal direction to a number of interpolation pixels in a vertical direction is an integer multiple of a ratio of a number of pixels in a horizontal direction of the image sensor to a number of pixels in a vertical direction of the image sensor, where the multiple is the integer is at least 2.

13 . The method of claim 12 , wherein the first image is a left image, and the second image is a right image, and wherein the first image and the second image are captured by the image sensor at the same time.

14 . The method of claim 12 , wherein the first interpolated pixels are positioned in the first interpolated image based on positions of respective pairs of pixels in the first image, and wherein the second interpolated pixels are positioned in the second interpolated image based on positions of respective pairs of pixels in the second image.

15 . The method of claim 14 , wherein the pairs of pixels correspond to pairs of the dual-pixels of the image sensor.

16 . The method of claim 15 , wherein each pixel of the first interpolated image has a position therein that corresponds to a position between the pixels of a corresponding pair of pixels of the first image.

17 . The method of claim 14 , wherein the first interpolated pixels comprise respective averages of the pixel values in the first image.

18 . The method of claim 14 , wherein the generating of the first and second interpolated pixels comprises:

performing, by an adder, additions of the pixel values in the first image and additions of the pixel values in the second image; and

performing, by a shifter, bit-shift operations on results of the additions outputted from the adder.

19 . The method of claim 14 , wherein the generating of the first interpolated image and the second interpolated image comprises performing interpolation such that a number of pixels in a horizontal direction in each of the first image and the second image is the same as a number of pixels in a vertical direction in each of the first image and the second image.

20 . The method of claim 14 , wherein the generating of the first interpolated image and the second interpolated image comprises performing interpolation such that an image aspect ratio of each of the first image and the second image is the same as, or a multiple of, an image aspect ratio of the image sensor.

21 . A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 12 .

22 . A method comprising:

accessing a first image comprising first pixels, wherein each first pixel comprises a first value, wherein the first image is generated by an image sensor comprising pixels, wherein the pixels of the image sensor comprise dual-pixels and non-dual-pixels, each dual-pixel respectively comprising first and second photodiodes, wherein the first values are provided by the respective first photodiodes; and

generating a first interpolated image by interpolating pairs of first photodiodes of different dual-pixels,

wherein the processor is further configured to perform interpolation such that a ratio of a number of interpolation pixels in a horizontal direction to a number of interpolation pixels in a vertical direction is an integer multiple of a ratio of a number of pixels in a horizontal direction of the image sensor to a number of pixels in a vertical direction of the image sensor, where the integer multiple is at least 2.

23 . The method of claim 22 , wherein each first photodiode has a different point spread function than its corresponding second photodiode.

24 . The method of claim 22 , wherein the first photodiodes are left photodiodes and the second photodiodes are right photodiodes.

25 . The method of claim 22 , wherein each dual-pixel further comprises a lens configured to refract light to the corresponding first and second photodiodes.

26 . The method of claim 24 , further comprising:

accessing a second image comprising second pixels, wherein each second pixel comprises a second value, wherein the second image is generated by the image sensor concurrently with the first image, and wherein the second values are provided by the respective second photodiodes; and

generating a second interpolated image by interpolating pairs of the second pixels.

27 . The method of claim 26 , further comprising generating a disparity map based on the first interpolated image and the second interpolated image.

28 . The method of claim 27 , further comprising autofocusing a camera based on the disparity map.

29 . The apparatus of claim 1 , wherein the processor comprises:

an adder configured to perform addition of the first pixel value and the second pixel value; and

a shifter configured to perform a division operation by performing a bit-shift operation on a result of the addition that is outputted from the adder,

wherein each of the first image and second image comprises an initial value represented by a first number of bits, and wherein the initial value, after storage, is represented by a second number of bits higher than the first number of bits as the respective first and second pixel values.

30 . The method of claim 22 , wherein the dual-pixels comprise an initial value represented by a first number of bits, and wherein the initial value, after storage, is represented by a second number of bits higher than the first number of bits as the respective first values,

wherein the processor is further configured to perform addition of the first values via an adder; and

perform a division operation by performing a bit-shift operation on a result of the addition that is outputted from the adder.