Electronic device and method for controlling same
An electronic device is disclosed. The electronic device may comprise a first image sensor, a second image sensor, and a processor, wherein the processor may: acquire a first depth image and a confidence map by using the first image sensor; acquire an RGB image by using the second image sensor; acquire a second depth image on the basis of the confidence map and the RGB image; and acquire a third depth image by composing the first depth image and the second depth image on the basis of the pixel value of the confidence map.
1 . An electronic device, comprising:
a first image sensor;
a second image sensor; and
a processor configured to:
obtain a first depth image and a confidence map corresponding to the first depth image based on information received from the first image sensor,
obtain an RGB image corresponding to the first depth image based on information received from the second image sensor,
obtain a second depth image based on the confidence map and the RGB image, and
obtain a third depth image based on a composition of the first depth image and the second depth image determined based on a pixel value of the confidence map,
wherein the third depth image is obtained by combining the first depth image and the second depth image based on the pixel value of the confidence map.
2 . The electronic device as claimed in claim 1 , wherein the processor is further configured to obtain a grayscale image from the RGB image, and
the second depth image is obtained by performing stereo matching on the confidence map and the grayscale image.
3 . The electronic device as claimed in claim 2 , wherein the processor is further configured to obtain the second depth image by performing stereo matching on the confidence map and the grayscale image based on a shape of an object included in the confidence map and the grayscale image.
4 . The electronic device as claimed in claim 1 , wherein the processor is further configured to:
determine a first composition ratio value of the first depth image and a second composition ratio value of the second depth image based on the pixel value of the confidence map, and
obtain the third depth image by combining the first depth image and the second depth image based on the first composition ratio value and the second composition ratio value.
5 . The electronic device as claimed in claim 4 , wherein the processor is further configured to:
determine the first composition ratio value of the first depth image to be greater than the second composition ratio value of the second depth image for a first region in which a pixel value is greater than a reference value among a plurality of regions of the confidence map, and
determine the first composition ratio value to be smaller than the second composition ratio value for the region in which the pixel value is smaller than the reference value among a plurality of regions of the confidence map.
6 . The electronic device as claimed in claim 1 , wherein the processor is further configured to:
obtain a depth value of the second depth image as a depth value of the third depth image for a first region in the third depth image corresponding to a first region among a plurality of regions of the first depth image, in which a depth value of the first depth image is smaller than a first threshold distance, and
obtain a depth value of the first depth image as a depth value of the third depth image for a second region in the third depth image corresponding to a second region among a plurality of regions of the first depth image in which a depth value of the first depth image is greater than a second threshold distance.
7 . The electronic device as claimed in claim 1 , wherein the processor is further configured to:
identify an object included in the RGB image,
identify each region of the first depth image and the second depth image corresponding to the identified object, and
obtain the third depth image by combining the first depth image and the second depth image based on a composition ratio for each of the regions.
8 . The electronic device as claimed in claim 1 , wherein the first image sensor is a time of flight (ToF) sensor, and
the second image sensor is an RGB sensor.
9 . A method for controlling an electronic device, comprising: obtaining a first depth image and a confidence map corresponding to the first depth image based on information received from a first image sensor; obtaining an RGB image corresponding to the first depth image based on information received from a second image sensor; obtaining a second depth image based on the confidence map and the RGB image; and obtaining a third depth image based on a composition of the first depth image and the second depth image determined based on a pixel value of the confidence map wherein the third depth image is obtained by combining the first depth image and the second depth image based on the pixel value of the confidence map.
10 . The method as claimed in claim 9 , further comprises: obtaining a grayscale image for the RGB image, and
obtaining the second depth image by stereo matching the confidence map and the grayscale image.
11 . The method as claimed in claim 10 , further comprises: obtaining the second depth image by stereo matching the confidence map and the grayscale image based on a shape of an object included in the confidence map and the grayscale image.
12 . The method as claimed in claim 9 , further comprises:
determining a first composition ratio value of the first depth image and a second composition ratio value of the second depth image based on the pixel value of the confidence map, and
obtaining the third depth image by combining the first depth image and the second depth image based on the first composition ratio value and the second composition ratio value.
13 . The method as claimed in claim 12 , further comprising:
determining the first composition ratio value of the first depth image to be greater than the second composition ratio value of the second depth image for a first region in which a pixel value is greater than a reference value among a plurality of regions of the confidence map, and
determining the first composition ratio value to be smaller than the second composition ratio value for the region in which the pixel value is smaller than the reference value among a plurality of regions of the confidence map.
14 . The method as claimed in claim 9 , further comprises:
obtaining a depth value of the second depth image as a depth value of the third depth image for a first region in the third depth image corresponding to a first region among a plurality of regions of the first depth image, in which a depth value of the first depth image is smaller than a first threshold distance, and
obtaining a depth value of the first depth image as a depth value of the third depth image for a second region in the third depth image corresponding to a second region among a plurality of regions of the first depth image in which a depth value of the first depth image is greater than a second threshold distance.
15 . The method as claimed in claim 9 , further comprises:
identifying an object included in the RGB image;
identifying each region of the first depth image and the second depth image corresponding to the identified object, and
acquiring the third depth image by combining the first depth image and the second depth image based on a composition ratio for each of the identified regions.
16 . The electronic device as claimed in claim 1 , wherein the third depth image is obtained based on a first composition ratio of the first depth image and a second composition ratio of the second depth image determined based on the pixel value of the confidence map.
17 . The electronic device as claimed in claim 1 , wherein a depth value of each pixel of the second depth image is a distance from the electronic device to an object corresponding to the each pixel of the second depth image.