Electronic device and control method thereof
An electronic device and a control method thereof are provided. The electronic device includes a camera, a camera flash, and at least one processor configured to control the camera to capture a natural light image and a depth image of an object, control the camera and the camera flash to capture an artificial light image of the object, obtain distance information from the depth image to generate a depth mask image, create a cluster mask image from the natural light image, obtain a flash image in which the illuminance of the natural light image has been removed from the illuminance of the artificial light image, obtain an optimization parameter based on the distance information, the depth mask image, the cluster mask image, and the flash image, and obtain three-dimensional topographic information and surface reflection information about the object based on the obtained optimization parameter.
1. An electronic device comprising:
a camera;
a camera flash; and
at least one processor configured to:
control the camera to capture a natural light image and a depth image of an object,
control the camera and the camera flash to capture an artificial light image of the object comprising an artificial lighting,
obtain distance information from the depth image and generate a depth mask image,
generate a cluster mask image from the natural light image,
obtain a flash image in which illuminance of the natural light image has been removed from the illuminance of the artificial light image,
obtain an optimization parameter based on the distance information, the depth mask image, the cluster mask image, and the flash image, and
obtain three-dimensional (3D) geometry and surface reflection information about the object based on the obtained optimization parameter.
2. The electronic device of claim 1 , further comprising:
a display to display an image comprising a 3D object based on the obtained 3D geometry and surface reflection information of the object.
3. The electronic device of claim 1 , wherein the at least one processor is further configured to generate an image pyramid comprising a plurality of images by downsampling the depth mask image, cluster mask image, and flash image to a preset size.
4. The electronic device of claim 3 , wherein the at least one processor is further configured to:
obtain a first optimization parameter from the smallest image among the generated image pyramid,
obtain a second optimization parameter from the image of the next size based on the obtained first optimization parameter, and
obtain the optimization parameter from the largest image based on obtained n−1 optimization parameter.
5. The electronic device of claim 1 , wherein the at least one processor is further configured to:
set a pixel comprising distance information from among the depth images to 1, and
set a pixel having no distance information to 0 based on the distance information so as to generate the depth mask image.
6. The electronic device of claim 1 , wherein the at least one processor is further configured to:
identify a super-pixel by filtering the natural light image, and
generate the cluster mask image based on the identified super-pixel.
7. The electronic device of claim 6 , wherein the at least one processor is further configured to:
set the super-pixel to an independent cluster,
generate a weight graph between clusters including a node representing each cluster and an edge representing an average color distance of an area in which a plurality of clusters are adjacent,
obtain an edge weight by applying a weight to a difference in average color values between super-pixels included in and each adjacent cluster based on adjacent length between the clusters,
obtain an average color value of the cluster, and
based on the obtained average color value being less than or equal to a preset color threshold and the edge weight being less than or equal to the preset weight threshold value, generate the cluster mask image by merging clusters.
8. The electronic device of claim 1 , wherein the optimization parameter comprises depth change, change angle of a normal vector, diffuse albedo, specular albedo and roughness information per cluster.
9. The electronic device of claim 8 , wherein the at least one processor is further configured to obtain the 3D geometry based on the depth change and the change angle of the normal vector.
10. The electronic device of claim 8 , wherein the at least one processor is further configured to obtain the surface reflection information based on the diffuse albedo, specular albedo and roughness information per cluster.
11. A method of controlling an electronic device, the method comprising:
capturing a natural light image and a depth image of an object;
capturing an artificial light image of the object comprising an artificial lighting;
obtaining distance information from the depth image and generating a depth mask image;
generating a cluster mask image from the natural light image;
obtaining a flash image in which illuminance of the natural light image has been removed from the illuminance of the artificial light image;
obtaining an optimization parameter based on the distance information, the depth mask image, the cluster mask image, and the flash image; and
obtaining three-dimensional (3D) geometry and surface reflection information about the object based on the obtained optimization parameter.
12. The method of claim 11 , further comprising:
displaying an image comprising a 3D object based on the obtained 3D geometry and surface reflection information of the object.
13. The method of claim 11 , wherein the obtaining of the optimization parameter comprises generating an image pyramid comprising a plurality of images by downsampling the depth mask image, cluster mask image, and flash image to a preset size.
14. The method of claim 13 , wherein the obtaining of the optimization parameter comprises:
obtaining a first optimization parameter from the smallest image among the generated image pyramid;
obtaining a second optimization parameter from the image of the next size based on the obtained first optimization parameter; and
obtaining the optimization parameter from the largest image based on obtained n−1 optimization parameter.
15. The method of claim 11 , wherein the generating of the depth mask image comprises:
setting a pixel comprising distance information from among the depth images to 1; and
setting a pixel having no distance information to 0 based on the distance information so as to generate the depth mask image.