Method and apparatus for converting depth image in high resolution

1. A method, comprising:

increasing, using a processor, a depth image resolution of a depth image to match a color image resolution of a two dimensional color image of a same scene and to produce an up-scaled depth image;

removing, using the processor, noise in the up-scaled depth image using the color image and the up-scaled depth image, the removing including updating a depth of a target pixel in the up-scaled depth image based on similarities of neighboring pixels with respect to the target pixel and depths of the neighboring pixel, the similarities of the neighboring pixels being based on differences in one or more attribute values between the neighboring pixels and the target pixel;

sharpening, using the processor, a boundary of an object in the up-scaled depth image from which the noise is removed to produce sharpened depth image; and

generating, using the processor, a three dimensional color model of the scene using the sharpened depth image and the two dimensional color image.

2. The method of claim 1 , wherein the sharpening adjusts a pixel depth of a boundary pixel to a closest one of a maximum and a minimum pixel depth of neighboring pixels in the boundary.

3. The method of claim 1 , wherein

the neighboring pixels correspond to pixels, included in a window of which a center is the target pixel, from among pixels in the up-scaled depth image, and

the differences in the one or more attribute values including at least one of,

a first difference between a color of a pixel of the color image corresponding to the neighboring pixel and a color of a pixel of the color image corresponding to the target pixel,

a second difference between a depth of the neighboring pixel and a depth of the target pixel, and

a third difference between a location of the neighboring pixel and a location of the target pixel.

4. A method of processing a depth image and a color image associated with a scene, the method comprising:

increasing, using a processor, a depth image resolution of the depth image by up-scaling the depth image using interpolation to produce an up-scaled depth image;

removing, using the processor, noise in the up-scaled depth image using the color image and the up-scaled depth image, the removing including updating a depth of a target pixel in the up-scaled depth image based on similarities of neighboring pixels with respect to the target pixel and depths of the neighboring pixels, the similarities of the neighboring pixels being based on differences in one or more attribute values between the neighboring pixels and the target pixel; and

sharpening, using the processor, a boundary between objects in the upscaled depth image from which the noise is removed,

wherein the depth image resolution of the up-scaled depth image matches a color image resolution of the color image.

5. The method of claim 4 , wherein

the neighboring pixels correspond to pixels included in a window of which a center is the target pixel, from among pixels in the up-scaled depth image, and

the differences in the one or more attribute values including at least one of,

a first difference between a color of a pixel of the color image corresponding to the neighboring pixel and a color of a pixel of the color image corresponding to the target pixel,

a second difference between a depth of the neighboring pixel and a depth of the target pixel, and

a third difference between a location of the neighboring pixel and a location of the target pixel.

6. The method of claim 5 , wherein the updating updates the depth of the target pixel to a value obtained by dividing a first result value by a second result value, wherein

the first result value is a sum of products of the depths of the neighboring pixels and the similarities of the neighboring pixels with respect to the target pixel, and

the second result value is a sum of the similarities of the neighboring pixels with respect to the target pixel.

7. The method of claim 5 , wherein

the pixels in the up-scaled depth image are sequentially selected as the target pixel and updated, and

the target pixel is repeatedly updated, at least once, based on previously updated depths of neighboring pixels.

8. The method of claim 5 , wherein the updating updates the target pixel within a range between a minimum value and a maximum value, the minimum value being a minimum depth from among depths of the neighboring pixels before updating, and the maximum value being a maximum depth from among the depths of neighboring pixels before updating.

9. The method of claim 5 , further comprising:

generating, using the processor, the color image by down-sampling an original color image associated with the scene to match the depth image resolution of the depth image,

wherein the increasing of the depth image resolution, the generating of the color image, and the removing of the noise are repeatedly performed until the depth image resolution of the up-scaled depth image matches an original resolution of the original color image.

10. The method of claim 4 , wherein the sharpening further comprises:

identifying a maximum depth and a minimum depth from among depths of neighboring pixels of a target pixel in the up-scaled depth image;

determining whether the target pixel exists in a boundary area between objects in the up-scaled depth image; and

replacing a depth of the target pixel with a replacement depth that is closer to the depth of the target pixel between the maximum depth and the minimum depth, if the target pixel exists in the boundary area,

wherein the neighboring pixels correspond to pixels included in a window of which a center is the target pixel, from among pixels in the up-scaled depth image.

11. The method of claim 10 , wherein the determining includes determining if a difference between the maximum depth and the minimum depth is greater than a threshold value.

12. A non-transitory computer-readable medium comprising computer-readable instructions for instructing a computer to perform the method of claim 4 .

13. An apparatus for processing a depth image and a color image associated with a scene, the apparatus comprising:

a processor; and

a memory containing computer-readable code that, when executed by the processor, causes the processor to,

perform up-scaling of the depth image using interpolation to increase a depth image resolution of the depth image to produce an up-scaled depth image,

remove noise in the up-scaled depth image using the color image and the up-scaled depth image by updating a depth of a target pixel in the up-scaled depth image based on similarities of neighboring pixels with respect to the target pixel and depths of the neighboring pixels, the similarities of the neighboring pixels being based on differences in one or more attribute values between the neighboring pixels and the target pixel, and

sharpen a boundary between objects in the up-scaled depth image from which the noise is removed,

wherein the depth image resolution of the up-scaled depth image matches a color image resolution of the color image.

14. The apparatus of claim 13 , wherein

the neighboring pixels correspond to pixels, included in a window of which a center is the target pixel, from among pixels in the up-scaled depth image, and

the differences in the one or more attribute values including at least one of,

a first difference between a color of a pixel of the color image corresponding to the neighboring pixel and a color of a pixel of the color image corresponding to the target pixel,

a second difference between a depth of the neighboring pixel and a depth of the target pixel, and

a third difference between a location of the neighboring pixel and a location of the target pixel.

15. The apparatus of claim 14 , wherein the processor is configured to update the depth of the target pixel to a value obtained by dividing a first result value by a second result value,

wherein,

the first result value is a sum of products between the depths of the neighboring pixels and the similarities of the neighboring pixels with respect to the target pixel, and

the second result value is a sum of the similarities of the neighboring pixels with respect to the target pixel.

16. The apparatus of claim 14 , wherein

the processor is configured to,

sequentially select the pixels in the up-scaled depth image as the target pixel and update the target pixel, and

repeatedly update the target pixel, at least once, based on previously updated depths of neighboring pixels.

17. The apparatus of claim 14 , wherein the processor is configured to update the target pixel within a range between a minimum value and a maximum value, the minimum value being a minimum depth from among depths of the neighboring pixels before updating, and the maximum value being a maximum depth from among the depths of the neighboring pixels before updating.

18. The apparatus of claim 14 , wherein the processor is further configured to generate the color image by down-sampling an original color image associated with the scene to match the depth image resolution of the depth image,

wherein the processor repeatedly operates until the depth image resolution of the up-scaled depth image matches an original resolution of the original color image.

19. The apparatus of claim 13 , wherein, the processor is configured to sharpen the boundary between the objects by,

identifying a maximum depth and a minimum depth from among depths of neighboring pixels of a target pixel in the up-scaled depth image,

determining whether the target pixel exists in a boundary area between objects in the up-scaled depth image, and

replacing a depth of the target pixel with a replacement depth that is closer to the depth of the target pixel from among the maximum depth and the minimum depth if the target pixel exists in the boundary area, wherein

the neighboring pixels correspond to pixels, included in a window of which a center is the target pixel, from among pixels in the up-scaled depth image.

20. The apparatus of claim 19 , wherein the processor is configured to determine that the target pixel exists in the boundary area if a difference between the maximum depth and the minimum depth is greater than a threshold value.