Method and system for real-time motion artifact handling and noise removal for ToF sensor images

1. A method for real-time motion artifact handling and noise removal for time-of-flight (ToF) sensor images, the ToF sensor images corresponding to received signals (r(t)) received by a ToF sensor following sending of modulated sent signals (s(t)), the method comprising:

calculating values of a cross correlation function c(τ) at a plurality of temporally spaced positions or phases from the sent (s(t)) and received (r(t)) signals, thereby deriving a plurality of respective cross correlation values [c(τ 0 ), c(τ 1 ), c(τ 2 ), c(τ 3 )], wherein each of the plurality of respective cross correlation values [c(τ 0 ), c(τ 1 ), c(τ 2 ), c(τ 3 )] corresponds to a respective one of a plurality of phase-shifted images;

deriving, from the plurality of cross correlation values [c(τ 0 ), c(τ 1 ), c(τ 2 ), c(τ 3 )], a depth map D, the depth map D having values representing, for each pixel, distance to a portion of an object upon which the sent signals (s(t)) are incident;

selecting, from the plurality of phase-shifted images, a guidance image (I; I′), the guidance image (I; I′) being an image having well defined edges; and

generating an output image D′ based on the depth map D and the guidance image (I; I′), the output image D′ comprising an edge-preserving and smoothed version of depth map D, the edge-preserving being from guidance image (I; I′).

2. The method of claim 1 , comprising acquiring the plurality of phase shifted images in succession, each phase shifted image corresponding to a respective temporally spaced position or phase.

3. The method of claim 1 , wherein selecting the guidance image comprises selecting as the guidance image a phase-shifted image from a plurality of previously-acquired phase-shifted images, based on one or more predetermined criteria.

4. The method of claim 3 , wherein the predetermined criteria comprise that the phase-shifted image (i) with maximum amplitude of the object degraded by motion artefact, (ii) with maximum object edge sharpness value, (iii) with the best edge contrast and/or image SNR, (iv) with the maximum average spatial amplitude or (v) that is non-corrupted, is selected as the guidance image.

5. The method of claim 1 , including using a guided filter (GF) to apply valid depth measurements to previously identified corrupted depth pixels due to motion artifacts.

6. The method of claim 1 , wherein generating an output image D′ comprises determining the output image D′ as:

D′ i =ā l I i + b l

where

a

l

_

=

1



w



⁢

∑

k

∈

w

i

⁢

⁢

a

k

,

b

l

_

=

1



w



⁢

∑

k

∈

w

i

⁢

⁢

b

k

,

⁢

with

⁢

⁢

a

k

=

1



w



⁢

∑

i

∈

w

k

⁢

⁢

I

i

⁢

D

i

-

I

_

k

⁢

⁢

D

_

k

σ

k

2

+

ɛ

with

⁢

⁢

b

k

=

D

_

k

-

a

k

⁢

I

_

k

,

are linear coefficients assumed to be constant in w k , Ī k and σ 2 k are respectively the mean and the variance of I in w k , |w| is the number of pixels in w k and ε is a regularization parameter penalizing large

a

k

·

D

k

_

=

1



w



⁢

∑

i

∈

w

k

⁢

D

i

is the mean of D in w k .

7. The method of claim 1 , wherein generating an output image D′ comprises:

filtering the guidance image I to generate a de-noised guidance image I′; and

generating an output image D′ based on the depth map D and the de-noised guidance image I′.

8. The method of claim 7 , wherein filtering the guidance image I to generate a de-noised guidance image I′ comprises using a guided filter to perform said filtering.

9. The method of claim 1 , wherein generating an output image D′ further comprises:

generating a plausibility map P based on the plurality of cross correlation values [c(τ 0 ), c(τ 1 ), c(τ 2 ), c(τ 3 )], the plausibility map P comprising, for each pixel of the depth map D, a value indicative of whether the pixel is corrupted by motion or saturation; and

generating the output image D′ based on the depth map D and the plausibility map P, and on either the guidance image I or the de-noised guidance image I′.

10. The method of claim 9 , wherein generating the plausibility map P comprises determining, for each pixel, a metric p i representing the deviation of the phase-shifted amplitudes from a sinusoidal model of the cross-correlation function.

11. The method of claim 10 , wherein the metric p i comprises:

p i =|c (τ 1 )− c (τ 0 ) 31 c (τ 2 )+ c (τ 3 )|/( a +α)

where α is a regularization parameter preventing high value of p i when the amplitude a is low.

12. The method of claim 10 , wherein the plausibility map P has values P i , for each pixel i such that:

P

i

=

{

1

if

⁢

⁢

metric

⁢

⁢

p

i

>

δ

0

otherwise

where δ is a motion threshold value, above which motion is detected at pixel i.

13. The method of claim 12 , wherein δ is determined by capturing by the ToF sensor of an empty or motionless scene.

14. The method of claim 7 , wherein filtering the guidance image I to derive the de-noised guidance image I′ comprises:

applying an edge preserving de-noising filter to guidance image I, whereby edge information and noise reduction from the guidance image I is transferred to the output image D′.

15. The method of claim 7 , wherein filtering the guidance image I comprises deriving de-noised guidance image I′ using:

I′ i =ā l I i + b l .

where

a

_

i

=

1



w



⁢

∑

k

∈

w

i

⁢

a

k

,

with

⁢

⁢

a

k

=

σ

k

2

σ

k

2

+

ɛ

b

_

i

=

1



w



⁢

∑

k

∈

w

i

⁢

b

k

,

with

⁢

⁢

b

k

=

(

1

-

a

k

)

·

I

k

_

,

σ k 2 is the variance of I in w k ,

|w| is the number of pixels in w k ,

ϵ is a regularization parameter penalizing large a k , and

I

k

_

=

1



w



⁢

∑

i

∈

w

k

⁢

I

i

is the mean of I in w k .

16. The method of claim 7 , wherein generating an output image D′ further comprises:

generating a plausibility map P based on the plurality of cross correlation values [c(τ 0 ), c(τ 1 ), c(τ 2 ), c(τ 3 )], the plausibility map P comprising, for each pixel of the depth map D, a value indicative of whether the pixel is corrupted by motion or saturation; and

generating the output image D′ based on the depth map D and the plausibility map P, and on either the guidance image I or the de-noised guidance image I′,

wherein generating an output image D′ comprises generating an output image D′ according to:

D′ i =ā l I′ i + b l .

where

a

_

i

=

1



w



⁢

∑

k

∈

w

i

⁢

1



w

k



⁢

∑

j

∈

w

k

⁢

I

j

′

⁢

D

j

⁢

P

j

-

I

_

k

⁢

D

_

k

σ

k

2

+

ɛ

,

⁢

b

_

i

=

1



w



⁢

∑

k

∈

w

i

⁢

D

_

k

-

a

k

⁢

I

_

k

,

and

D

_

k

=

1



w

k



⁢

∑

i

∈

w

k

⁢

D

i

⁢

P

i

is the mean of D in w k weighted by the map P,

|w| is the constant number of pixels in the window w i centered at pixel i,

|w k |=Σ i∈w k P i is the number of pixels in w k where the plausibility is 1, and

σ

k

2

=

1



w

k



⁢

∑

i

∈

w

k

⁢

I

i

′2

⁢

P

i

′

-

I

_

k

2

where

I

_

k

=

1



w

k



⁢

∑

i

∈

w

k

⁢

I

i

′

⁢

P

i

.

17. The method of claim 1 , wherein the output image D′ comprise a version of depth map D alternatively or additionally from which motion artifacts and/or noise have been suppressed or removed.

18. The method of claim 1 , wherein the cross correlation function c(τ) is calculated as:

c

⁡

(

τ

)

=

∫

t

=

0

T

⁢

r

⁡

(

t

)

·

s

⁡

(

t

+

τ

)

⁢

dt

=

h

+

a

2

⁢

cos

⁡

(

ωτ

+

φ

)

,

where

s(t) is the sent signal given by s(t)=1+cos(ωt),

r(t) is the received signal given by r(t)=h+a·cos(ωt−φ)

ω=2πf m is the angular modulation frequency,

f m is the modulation frequency, and

h is the background light plus the non-modulated part of the sent signal.

19. The method of claim 18 , wherein the cross correlation values [c(τ 0 ), c(τ 1 ), c(τ 2 ), c(τ 3 )] are determined from c(τ) as four samples (τ k ), k=0, . . . , 3, taken at four subsequent time intervals

τ

k

=

k

·

T

4

=

k

/

4

⁢

f

m

within a modulated period T.

20. The method of claim 10 , wherein the distance measurements d for each pixel of the depth map D are obtained from

d

=

L

2

⁢

π

·

ϕ

where

ϕ

=

atan

⁡

(

c

⁡

(

τ

3

)

-

c

⁡

(

τ

1

)

c

⁡

(

τ

0

)

-

c

⁡

(

τ

2

)

)

,

c≅3·10 8 m/s, the speed of light,

L is the working range or non-ambiguity distance range of the ToF sensor, and

L

=

c

2

⁢

f

m

.

21. The method of claim 1 , wherein an amplitude image A defined as A=[a ij ] m×n , where the a ij are determined from:

a= ½√{square root over (( c (τ 3 )− c (τ 1 )) 2 +( c (τ 0 )− c (τ 2 )) 2 )},

where c(τ 0 ), c(τ 1 ), c(τ 2 ), c(τ 3 ) are the cross correlation values.

22. The method of claim 1 , wherein four cross correlation values [c(τ 0 ), c(τ 1 ), c(τ 2 ), c(τ 3 )] are calculated from the cross correlation function c(τ).

23. A programmable image processing system when suitably programmed for carrying out the method of claim 1 , the system comprising circuitry for receiving or storing the received signals (r(t)) and sent signals (s(t)), and processing circuitry for performing the methods.

24. A system for real-time motion artifact handling and noise removal for time-of-flight (ToF) sensor images, the ToF sensor images corresponding to received signals (r(t)) received by a ToF sensor following sending of modulated sent signals (s(t)), the system comprising:

circuitry for receiving or storing the received signals (r(t)) and sent signals (s(t));

processing circuitry, coupled to the circuitry for receiving or storing the received signals (r(t)) and sent signals (s(t)), the processing circuitry being operable for

calculating values of a cross correlation function c(τ) at a plurality of temporally spaced positions or phases from the sent (s(t)) and received (r(t)) signals, thereby deriving a plurality of respective cross correlation values [c(τ 0 ), c(τ 1 ), c(τ 2 ), c(τ 3 )], wherein each of the plurality of respective cross correlation values [c(τ 0 ), c(τ 1 ), c(τ 2 ), c(τ 3 )] corresponds to a respective one of a plurality of phase-shifted images;

deriving, from the plurality of cross correlation values [c(τ 0 ), c(τ 1 ), c(τ 2 ), c(τ 3 )], a depth map D, the depth map D having values representing, for each pixel, distance to a portion of an object upon which the sent signals (s(t)) are incident;

selecting, from the plurality of phase-shifted images, a guidance image (I; I′), the guidance image (I; I′) being an image having well defined edges; and

generating an output image D′ based on the depth map D and the guidance image (I; I′), the output image D′ comprising an edge-preserving and smoothed version of depth map D, the edge-preserving being from guidance image (I; I′).

25. A non-transitory recordable, rewritable or storable medium having recorded or stored thereon data defining or transformable into instructions for execution by processing circuitry and corresponding to at least the steps of claim 1 .

26. A server computer incorporating a communications device and a memory device and being adapted for transmission on demand or otherwise of data defining or transformable into instructions for execution by processing circuitry and corresponding to at least the steps of claim 1 .