Method of depth-based imaging using an automatic trilateral filter for 3D stereo imagers
View Patent ↗A system of stereo imagers, including image processing units and methods of blurring an image, is presented. The image is received from an image sensor. For each pixel of the image, a depth filter component is determined based on a focal area of the image and a depth map associated with the image. For each pixel of the image, a trilateral filter is generated that includes a spatial filter component, a range filter component and the depth filter component. The respective trilateral filter is applied to corresponding pixels of the image to blur the image outside of the focal area. A refocus area or position may be determined by imaging geometry or may be selected manually via a user interface.
1. A method of blurring an image comprising:
receiving the image from an image sensor;
determining, for each pixel of the image, a depth filter component based on a focal area of the image and a depth map associated with the image;
generating, for each pixel of the image, a trilateral filter including a spatial filter component, a range filter component and the depth filter component; and
applying the respective trilateral filter to corresponding pixels of the image to blur the image outside of the focal area, wherein the trilateral filter for a current pixel position is based on a sum of the product of a plurality of different filter functions over a neighborhood of the current pixel position.
2. The method according to claim 1 , wherein the applying of the respective trilateral filter sharpens pixels within the focal area.
3. The method according to claim 1 , the method further including: determining the focal area based on an imaging geometry of the image sensor.
4. The method according to claim 1 , the method further including: receiving a selection indicating the focal area from a user interface.
5. The method according to claim 1 , wherein the image sensor includes first and second image sensors, the receiving of the image includes receiving first and second images from the respective first and second image sensors, the method further including:
extracting the depth map from the first and second images, one of the first and second images being selected as the image.
6. The method according to claim 1 , wherein, for each pixel, the spatial filter component is determined based on a range of neighboring pixels relative to the pixel.
7. The method according to claim 1 , wherein, for each pixel, the range filter component is determined based on pixel intensities of neighboring pixels relative to the pixel.
8. The method according to claim 1 , wherein the generating of the trilateral filter includes applying a weight to each of the depth filter component, the spatial filter component and the range filter component.
9. The method according to claim 8 , wherein the weight of at least one of the depth filter component, the spatial filter component and the range filter component is different.
10. The method according to claim 1 , wherein a kernel size of the depth filter component includes a piecewise linear function having an absolute value of a slope relative to a disparity of the image that is greater than one outside of the focal area.
11. The method according to claim 10 , wherein the determining of the depth filter component includes adjusting the slope of the kernel size to control a degree of blurring by the depth filter component.
12. The method according to claim 1 , wherein the trilateral filter (TF p ), for a current pixel position p, is determined according to:
TF
p
=
1
W
p
∑
q
∈
S
G
σ
s
(
p
-
q
)
G
σ
r
(
I
p
-
I
q
)
G
σ
d
(
Z
p
-
Z
q
)
where G(.) is a filter function, S is a neighborhood of the current pixel position p, q is another pixel position within neighborhood S, I is a pixel intensity value, Z is a depth value, W p is a normalization factor, σ s is a space control parameter, σ r is a range control parameter and σ d is a depth control parameter.
13. The method according to claim 12 , wherein the trilateral filter includes a Dirac delta function when the depth value Z p of the current pixel position p is within the focal area.
14. The method according to claim 12 , wherein the trilateral filter includes a different filter function for at least one of the depth filter component, the spatial filter component or the range filter component.
15. An image processing unit for blurring an image comprising:
an input port configured to receive an image from an image sensor with a corresponding lens, wherein the image processing unit is configured to automatically determine a focal area from an imaging geometry of the lens; and
a blurring unit configured to:
generate, for each pixel of the image, a trilateral filter including a spatial filter component, a range filter component and a depth filter component, the depth filter component being determined based on the focal area of the image and a depth map associated with the image, and
apply the respective trilateral filter to corresponding pixels of the image to blur the image outside of the focal area.
16. The image processing unit according to claim 15 , wherein the image sensor includes first and second image sensors, and the image includes first and second images from the respective first and second image sensors,
the image processing unit further including a depth map extractor configured to extract the depth map from the first and second images, one of the first and second images being used as the image by the blurring unit.
17. The image processing unit according to claim 15 , wherein the blurring unit is configured to sharpen the image within the focal area.
18. The image processing unit according to claim 15 , wherein the image processing unit is configured to receive a selection indicating the focal area.
19. The image processing unit according to claim 15 , wherein the blurring unit is configured to control at least one of a weight or a filter function of each of the spatial filter component, the range filter component and the depth filter component.
20. The image processing unit according to claim 15 , wherein:
a kernel size of the depth filter component includes a piecewise linear function having an absolute value of a slope relative to a disparity of the image that is greater than one outside of the focal area, and
the blurring unit is configured to adjust the slope of the kernel size to control a degree of blurring by the depth filter component.