Robust localization and tracking of simultaneously moving sound sources using beamforming and particle filtering

1 . A system for localizing and tracking a plurality of sound sources, comprising:

a set of spatially spaced apart sound sensors to detect sound from the sound sources and produce corresponding sound signals;

a sound source detector responsive to the sound signals from the sound sensors and steered in a range of directions to localize the sound sources; and

a particle filtering tracker connected to the sound source detector for simultaneously tracking the plurality of sound sources.

2 . A sound source localizing and tracking system as defined in claim 1 , wherein the set of sound sensors comprises a predetermined number of omnidirectional microphones arranged in a predetermined array.

3 . A sound source localizing and tracking system as defined in claim 1 , wherein the sound source detector is a frequency-domain steered beamformer.

4 . A sound source localizing and tracking system as defined in claim 3 , wherein the steered beamformer comprises:

a calculator of sound power spectra and cross-power spectra of sound signal samples in overlapping windows;

a calculator of cross-correlations by averaging the cross-power spectra over a given period of time;

a calculator of an output energy of the steered beamformer from the calculated cross-correlations; and

a finder of a loudest sound source localized in a given direction, the given direction of the loudest sound source being found by maximizing the output energy of the steered beamformer.

5 . A sound source localizing and tracking system as defined in claim 4 , wherein the calculator of cross-correlations comprises:

a calculator for computing, in the frequency domain, whitened cross-correlations; and

a weighting function applied to the calculated whitened cross-correlations to act as a mask based on a signal-to-noise ratio.

6 . A sound source localizing and tracking system as defined in claim 5 , wherein the weighting function is modified to include a reverberation term in a noise estimate in order to make the system more robust to reverberation.

7 . A sound source localizing and tracking system as defined in claim 3 , wherein the steered beamformer produces an output energy and comprises:

a uniform triangular grid for the surface of a sphere to define directions;

a calculator of sound power spectra and cross-power spectra of sound signal samples in overlapping windows;

a calculator of cross-correlations by averaging the cross-power spectra over a given period of time;

a first algorithm for searching a best direction on the grid of the sphere;

a pre-computed table of time delays of arrival for each pair of sound sensors and each direction on the grid of the sphere; and

a finder of a loudest sound source in a direction of the grid of the sphere, the direction of the loudest sound source being found using the first algorithm and the pre-computed table by maximizing the output energy of the steered beamformer.

8 . A sound source localizing and tracking system as defined in claim 7 , further comprising a second algorithm for finding another sound source after having removed the contribution of the loudest sound source located by the finder.

9 . A sound source localizing and tracking system as defined in claim 7 , wherein the steered beamformer further comprises:

a refined grid for the surrounding of a point where a sound source was found in order to find a direction of localization of the found sound source with improved accuracy.

10 . A sound source localizing and tracking system as defined in claim 1 , wherein the particle filtering tracker models each sound source using a number of particles having respective directions and weights.

11 . A sound source localizing and tracking system as defined in claim 1 , wherein the particle filtering tracker comprises:

a calculator of a probability that a potential source is a real source.

12 . A sound source localizing and tracking system as defined in claim 1 , wherein the particle filtering tracker comprises:

a calculator of a probability that a real source corresponds to a potential source detected by the sound source detector.

13 . A sound source localizing and tracking system as defined in claim 10 , wherein the particle filtering tracker comprises:

a calculator of (a) at least one of a probability that a sound source is observed and a probability that a real sound source corresponds to a potential sound source, and (b) a probability density of observing a sound source at a given particle position; and

a calculator of updated particle weights in response to said probability density and said at least one probability.

14 . A sound source localizing and tracking system as defined in claim 1 , wherein the particle filtering tracker comprises:

an adder of a new source when a probability that the new source is real is higher than a first threshold.

15 . A sound source localizing and tracking system as defined in claim 14 , wherein the sound source localizing and tracking system assumes that the added new source exists if a probability of existence of said new source reaches a second threshold.

16 . A sound source localizing and tracking system as defined in claim 1 , wherein the particle filtering tracker comprises:

a subtractor of a source when the latter source has not been observed for a certain period of time.

17 . A sound source localizing and tracking system as defined in claim 13 , wherein the particle filtering tracker comprises:

an estimator of a position of each source as a weighted average of the positions of its particles, said estimator being responsive to the calculated, updated particle weights.

18 . A system for localizing at least one sound source, comprising:

a set of spatially spaced apart sound sensors to detect sound from said at least one sound source and produce corresponding sound signals; and

a frequency-domain beamformer responsive to the sound signals from the sound sensors and steered in a range of directions to localize, in a single step, said at least one sound source.

19 . A sound source localizing system as defined in claim 18 , wherein the set of sound sensors comprises a predetermined number of omnidirectional microphones arranged in a predetermined array.

20 . A sound source localizing system as defined in claim 18 , wherein the steered beamformer comprises:

a calculator of sound power spectra and cross-power spectra of sound signal samples in overlapping windows;

a calculator of cross-correlations by averaging the cross-power spectra over a given period of time;

a calculator of an output energy of the steered beamformer from the calculated cross-correlations; and

a finder of a loudest sound source localized in a given direction, the given direction of the loudest sound source being found by maximizing the output energy of the steered beamformer.

21 . A sound source localizing system as defined in claim 20 , wherein the calculator of cross-correlations comprises:

a calculator for computing, in the frequency domain, whitened cross-correlations; and

a weighting function applied to the calculated whitened cross-correlations to act as a mask based on a signal-to-noise ratio.

22 . A sound source localizing system as defined in claim 21 , wherein the weighting function is modified to include a reverberation term in a noise estimate in order to make the system more robust to reverberation.

23 . A sound source localizing and tracking system as defined in claim 18 , wherein the steered beamformer produces an output energy and comprises:

a uniform triangular grid for the surface of a sphere to define directions;

a calculator of sound power spectra and cross-power spectra of sound signal samples in overlapping windows;

a calculator of cross-correlations by averaging the cross-power spectra over a given period of time;

a first algorithm for searching a best direction on the grid of the sphere;

a pre-computed table of time delays of arrival for each pair of sound sensors and each direction on the grid of the sphere; and

a finder of a loudest sound source in a direction of the grid of the sphere, the direction of the loudest sound source being found using the first algorithm and the pre-computed table by maximizing the output energy of the steered beamformer.

24 . A sound source localizing system as defined in claim 23 , further comprising a second algorithm for finding another sound source after having removed the contribution of the loudest sound source located by the finder.

25 . A sound source localizing and tracking system as defined in claim 23 , wherein the steered beamformer further comprises:

a refined grid for the surrounding of a point where a sound source was found in order to find a direction of localization of the found sound source with improved accuracy.

26 . A system for tracking a plurality of sound sources, comprising:

a set of spatially spaced apart sound sensors to detect sound from the sound sources and produce corresponding sound signals; and

a sound source particle filtering tracker responsive to the sound signals from the sound sensors for simultaneously tracking the plurality of sound sources.

27 . A sound source tracking system as defined in claim 26 , wherein the particle filtering tracker models each sound source using a number of particles having respective directions and weights.

28 . A sound source tracking system as defined in claim 26 , wherein the particle filtering tracker comprises:

a calculator of a probability that a potential source is a real source.

29 . A sound source tracking system as defined in claim 26 , wherein the particle filtering tracker comprises:

a calculator of a probability that a real source corresponds to a potential source.

30 . A sound source tracking system as defined in claim 27 , wherein the particle filtering tracker comprises:

a calculator of (a) at least one of a probability that a sound source is observed and a probability that a real sound source corresponds to a potential sound source, and (b) a probability density of observing a sound source at a given particle position; and

a calculator of updated particle weights in response to said probability density and said at least one probability.

31 . A sound source tracking system as defined in claim 26 , wherein the particle filtering tracker comprises:

an adder of a new source when a probability that the new source is real is higher than a first threshold.

32 . A sound source tracking system as defined in claim 31 , wherein the sound source tracking system assumes that the added new source exists if a probability of existence of said new source reaches a second threshold.

33 . A sound source tracking system as defined in claim 26 , wherein the particle filtering tracker comprises:

a subtractor of a source when the latter source has not been observed for a certain period of time.

34 . A sound source tracking system as defined in claim 30 , wherein the particle filtering tracker comprises:

an estimator of a position of each source as a weighted average of the positions of its particles, said estimator being responsive to the calculated, updated particle weights.

35 . A method for localizing and tracking a plurality of sound sources, comprising:

detecting sound from the sound sources through a set of spatially spaced apart sound sensors to produce corresponding sound signals;

localizing the sound sources in response to the sound signals, localizing the sound sources including steering in a range of directions a sound source detector having an output; and

simultaneously tracking the plurality of sound sources, using particle filtering, in relation to the output from the sound source detector.

36 . A sound source localizing and tracking method as defined in claim 35 , wherein steering a sound source detector comprises steering a frequency-domain beamformer.

37 . A sound source localizing and tracking method as defined in claim 36 , wherein localizing the sound sources comprises:

computing sound power spectra and cross-power spectra of sound signal samples in overlapping windows;

computing cross-correlations by averaging the cross-power spectra over a given period of time;

computing an output energy of the steered beamformer from the calculated cross-correlations; and

finding a loudest sound source localized in a given direction, the given direction of the loudest sound source being found by maximizing the output energy of the steered beamformer.

38 . A sound source localizing and tracking method as defined in claim 37 , wherein computing the cross-correlations comprises:

computing, in the frequency domain, whitened cross-correlations; and

applying a weighting function to the computed whitened cross-correlations to act as a mask based on a signal-to-noise ratio.

39 . A sound source localizing and tracking method as defined in claim 38 , comprising modifying the weighting function by including a reverberation term in a noise estimate in order to make the method more robust to reverberation.

40 . A sound source localizing and tracking method as defined in claim 36 , wherein localizing the sound sources comprises:

defining a uniform triangular grid for the surface of a sphere to define directions;

computing sound power spectra and cross-power spectra of sound signal samples in overlapping windows;

computing cross-correlations by averaging the cross-power spectra over a given period of time;

pre-computing a table of time delays of arrival for each pair of sound sensors and each direction on the grid of the sphere; and

finding a loudest sound source in a direction of the grid of the sphere, finding the loudest sound source comprising searching a best direction on the grid of the sphere using a first algorithm and the pre-computed table by maximizing an output energy of the steered beamformer.

41 . A sound source localizing and tracking method as defined in claim 40 , comprising finding another sound source, using a second algorithm, after having removed the contribution of the located, loudest sound source.

42 . A sound source localizing and tracking method as defined in claim 40 , wherein localizing the sound sources further comprises:

defining a refined grid for the surrounding of a point where a sound source was found in order to find a direction of localization of the found sound source with improved accuracy.

43 . A sound source localizing and tracking method as defined in claim 35 , wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises modeling each sound source using a number of particles having respective directions and weights.

44 . A sound source localizing and tracking method as defined in claim 35 , wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises:

computing a probability that a potential source is a real source.

45 . A sound source localizing and tracking method as defined in claim 35 , wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises:

computing a probability that a real source corresponds to a potential source detected by the sound source detector.

46 . A sound source localizing and tracking method as defined in claim 43 , wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises:

computing (a) at least one of a probability that a sound source is observed and a probability that a real sound source corresponds to a potential sound source, and (b) a probability density of observing a sound source at a given particle position; and

computing updated particle weights in response to said probability density and said at least one probability.

47 . A sound source localizing and tracking method as defined in claim 35 , wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises:

adding a new source when a probability that the new source is real is higher than a first threshold.

48 . A sound source localizing and tracking method as defined in claim 47 , wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises assuming that the added new source exists if a probability of existence of said new source reaches a second threshold.

49 . A sound source localizing and tracking method as defined in claim 35 , wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises:

removing a sound source when the latter source has not been observed for a certain period of time.

50 . A sound source localizing and tracking method as defined in claim 43 , wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises:

estimating a position of each source as a weighted average of the positions of its particles, said estimator being responsive to the calculated, updated particle weights.

51 . A method for localizing at least one sound source, comprising:

detecting sound from said at least one sound source through a set of spatially spaced apart sound sensors to produce corresponding sound signals; and

localizing, in a single step, said at least one sound source in response to the sound signals, localizing said at least one sound source including steering a frequency-domain beamformer in a range of directions.

52 . A sound source localizing method as defined in claim 51 , wherein localizing, in a single step, said at least one sound source comprises:

computing sound power spectra and cross-power spectra of sound signal samples in overlapping windows;

computing cross-correlations by averaging the cross-power spectra over a given period of time;

computing an output energy of the steered beamformer from the calculated cross-correlations; and

finding a loudest sound source localized in a given direction, the given direction of the loudest sound source being found by maximizing the output energy of the steered beamformer.

53 . A sound source localizing method as defined in claim 52 , wherein computing the cross-correlations comprises:

computing, in the frequency domain, whitened cross-correlations; and

applying a weighting function to the computed whitened cross-correlations to act as a mask based on a signal-to-noise ratio.

54 . A sound source localizing method as defined in claim 53 , comprising modifying the weighting function by including a reverberation term in a noise estimate in order to make the method more robust to reverberation.

55 . A sound source localizing method as defined in claim 51 , wherein localizing, in a single step, said at least one sound source comprises:

defining a uniform triangular grid for the surface of a sphere to define directions;

computing sound power spectra and cross-power spectra of sound signal samples in overlapping windows;

computing cross-correlations by averaging the cross-power spectra over a given period of time;

pre-computing a table of time delays of arrival for each pair of sound sensors and each direction on the grid of the sphere; and

finding a loudest sound source in a direction of the grid of the sphere, finding the loudest sound source comprising searching a best direction on the grid of the sphere using a first algorithm and the pre-computed table by maximizing an output energy of the steered beamformer.

56 . A sound source localizing method as defined in claim 55 , comprising finding another sound source, using a second algorithm, after having removed the contribution of the located, loudest sound source.

57 . A sound source localizing method as defined in claim 55 , wherein localizing, in a single step, said at least one sound source further comprises:

defining a refined grid for the surrounding of a point where a sound source was found in order to find a direction of localization of the found sound source with improved accuracy.

58 . A method for tracking a plurality of sound sources, comprising:

detecting sound from the sound sources through a set of spatially spaced apart sound sensors to produce corresponding sound signals; and

simultaneously tracking the plurality of sound sources, using particle filtering responsive to the sound signals from the sound sensors.

59 . A sound source tracking method as defined in claim 58 , wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises modeling each sound source using a number of particles having respective directions and weights.

60 . A sound source tracking method as defined in claim 58 , wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises:

computing a probability that a potential source is a real source.

61 . A sound source tracking method as defined in claim 58 , wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises:

computing a probability that a real source corresponds to a potential source detected by the sound source detector.

62 . A sound source tracking method as defined in claim 59 , wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises:

computing (a) at least one of a probability that a sound source is observed and a probability that a real sound source corresponds to a potential sound source, and (b) a probability density of observing a sound source at a given particle position; and

computing updated particle weights in response to said probability density and said at least one probability.

63 . A sound source tracking method as defined in claim 58 , wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises:

adding a new source when a probability that the new source is real is higher than a first threshold.

64 . A sound source tracking method as defined in claim 63 , wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises assuming that the added new source exists if a probability of existence of said new source reaches a second threshold.

65 . A sound source tracking method as defined in claim 58 , wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises:

removing a sound source when the latter source has not been observed for a certain period of time.

66 . A sound source localizing and tracking method as defined in claim 59 , wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises:

estimating a position of each source as a weighted average of the positions of its particles, said estimator being responsive to the calculated, updated particle weights.