Method and apparatus for refining depth image

1. A method of refining a depth image, the method comprising:

extracting shading information of color pixels from a color image; and

refining a depth image corresponding to the color image based on surface normal information of an object included in the shading information, wherein the refining of the depth image includes distinguishing between a noise region and an edge region in the depth image based on a characteristic of a surface normal distribution represented in the color image and applying filters comprising different filter characteristics to the noise region and the edge region respectively to reduce a noise component while maintaining an edge characteristic substantially unchanged.

2. The method of claim 1 , further comprising extracting shading information of the color pixels based on depth information of the refined depth image.

3. The method of claim 1 , wherein the shading information corresponds to a vector dot product between the direction of the light source and the surface normal of the object surface.

4. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 .

5. The method of claim 1 , wherein the refining of the depth image comprises:

determining the filter characteristic of the filter to be applied to a current depth pixel, of the depth pixels, included in the depth image based on a surface normal distribution of the surface normal information for each of regions in the depth image; and

adjusting a depth value of the current depth pixel by applying the filter having the determined filter characteristic to the current depth pixel.

6. The method of claim 5 , wherein the determining of the filter characteristic comprises determining either one or both of a filter coefficient of the filter to be applied to the current depth pixel and a filter size of the filter to be applied to the current depth pixel based on the surface normal information.

7. The method of claim 1 , further comprising determining whether to refine a depth image of a current time based on depth information of a depth image of a previous time.

8. The method of claim 7 , further comprising refining the depth image of the current time based on the depth information of the depth image of the previous time in response to a difference between a color image corresponding to the depth image of the current time and a color image corresponding to the depth image of the previous time satisfying a preset condition.

9. A method of refining a depth image, the method comprising:

extracting shading information of color pixels from a color image; and

refining a depth image corresponding to the color image based on surface normal information of an object included in the shading information, wherein the refining of the depth image comprises:

determining a type of a region to which a current depth pixel included in the depth image belongs based on a surface normal distribution of the surface normal information in the region; and

adjusting a depth value of the current depth pixel by applying a filter corresponding to the determined type of the region to the current depth pixel.

10. The method of claim 9 , wherein the filter is configured to adjust the depth value of the current depth pixel based on a depth value of a neighboring depth pixel of the current depth pixel.

11. The method of claim 9 , wherein the determining of the type of the region comprises determining a region to which the current depth pixel belongs based on a change of surface normal values of neighboring pixels of the current depth pixel.

12. The method of claim 9 , wherein the determining of the type of the region comprises determining whether the current depth pixel belongs to a noise region, a surface region of the object, or an edge region of the object.

13. The method of claim 12 , wherein a filter size of a filter corresponding to the noise region or the surface region of the object is greater than a filter size of a filter corresponding to the edge region of the object.

14. The method of claim 12 , wherein a filter coefficient to be applied to a neighboring depth pixel of the current depth pixel varies depending on whether the filter to be applied to the current depth pixel is a filter corresponding to the noise region, a filter corresponding to the surface region of the object, or a filter corresponding to the edge region of the object.

15. A method of refining a depth image, the method comprising:

extracting shading information of color pixels from a color image; and

refining a depth image corresponding to the color image based on surface normal information of an object included in the shading information, further comprising extracting albedo information of the color pixels from the color image;

wherein the refining of the depth image comprises refining the depth image based on a first weight based on the surface normal information, a second weight based on the albedo information, and a third weight based on a difference between a color image of a current time and a color image of a previous time.

16. A depth image refining apparatus comprising:

a processor configured to:

extract shading information of color pixels from a color image, and

refine a depth image corresponding to the color image based on surface normal information of an object included in the shading information,

wherein the refining of the depth image includes distinguishing between a noise region and an edge region in the depth image based on a characteristic of a surface normal distribution represented in the color image and applying filters comprising different filter characteristics to the noise region and the edge region respectively to reduce a noise component while maintaining an edge characteristic substantially unchanged.

17. The depth image refining apparatus of claim 16 , wherein the processor is further configured to:

determine a filter characteristic of the filter to be applied to a current depth pixel, of the depth pixels, included in the depth image based on a surface normal distribution of the surface normal information for each of regions in the depth image, and

adjust a depth value of the current depth pixel by applying the filter having the determined filter characteristic to the current depth pixel.

18. The depth image refining apparatus of claim 16 , wherein the processor is further configured to:

determine a type of a region to which a current depth pixel, of the depth pixels, included in the depth image belongs based on a surface normal distribution of the surface normal information of the region, and

adjust a depth value of the current depth pixel by applying a filter corresponding to the determined type of the region to the current depth pixel.

19. The depth image refining apparatus of claim 16 , wherein the processor is further configured to:

extract albedo information of the color pixels from the color image, and

refine the depth image based on a first weight based on the surface normal information, a second weight based on the albedo information, and a third weight based on a difference between a color image of a current time and a color image of a previous time.

20. A method of refining a depth image, the method comprising:

determining a noise reducing method to be applied to a depth image based on surface normal information of an object in a color image corresponding to the depth image; and

refining the depth image by applying the determined noise reducing method to the depth image, the determined noise reducing method including distinguishing between a noise region and an edge region in the depth image based on a characteristic of a surface normal distribution represented in the color image and applying filters comprising different filter characteristics to the noise region and the edge region respectively to reduce a noise component while maintaining an edge characteristic substantially unchanged.

21. The method of claim 20 , wherein the color image is a color image of a current time; and

the method further comprises applying a temporal filter, as the filter, to the current depth pixel in response to a difference between a region of the color image of the current time corresponding to a region of the depth image in which the current depth pixel is located and a corresponding region of a color image of a previous time being less than a predetermined threshold.

22. The method of claim 20 , wherein

the determining of the noise reducing method comprises determining a filter characteristic of the filter based on the surface normal information; and

the refining of the depth image comprises applying the filter to a current depth pixel of the depth image.

23. The method of claim 22 , wherein the filter is a spatial filter comprising a filter coefficient to be applied to a depth value of the current depth pixel and filter coefficients to be applied to depth values of neighboring depth pixels of the current depth pixel to obtain a refined depth value of the current depth pixel; and

the determining of the filter characteristic comprises determining, based on the surface normal information, either one or both of a filter size of the spatial filter and a reduction rate at which the filter coefficients decrease from a center of the spatial filter to a periphery of the spatial filter.

24. The method of claim 23 , wherein the determining of the filter characteristic further comprises:

determining whether a region in which the current depth pixel is located is a noise region, a surface region of the object, or an edge region of the object based on a surface normal distribution of the surface normal information; and

determining either one or both of the filter size and the reduction rate based on the determined type of the region.

25. The method of claim 24 , wherein

the determining of the filter size comprises:

determining the filter size to be a first filter size in response to the determined type of the region being the noise region;

determining the filter size to be a second filter size smaller than the first filter size in response to the determined type of the region being the surface region of the object, and

determining the filter size to be a third filter size smaller than the second filter size in response to the determined type of the region being the edge region of the object; and

the determining of the reduction rate comprises:

determining the reduction rate to be a first reduction rate in response to the determined type of the region being the noise region;

determining the reduction rate to be a second reduction rate greater than the first reduction rate in response to the determined type of the region being the surface region of the object, and

determining the reduction rate to be a third reduction rate greater than the second reduction rate in response to the determined type of the region being the edge region of the object.