Object recognition using multi-modal matching scheme

1. A method performed by a device, the method comprising:

computing a plurality of acoustic-recognition features from audio recorded at a scene;

comparing the acoustic-recognition features to predetermined acoustic-recognition features corresponding to one or more objects to determine a sound source type of an object;

selecting keypoints corresponding to the object based on the sound source type;

and

identifying the object based on the selected keypoints and the sound source type.

2. The method of claim 1 , further comprising:

selecting one or more keypoint signatures corresponding to one or more objects, based on audio recorded at the scene;

identifying a plurality of keypoints in an image of the scene; and

comparing the keypoints to the keypoint signatures to identify the object.

3. The method of claim 1 , further comprising:

selecting a portion of a scene image based on the audio recorded at the scene; and

selecting the keypoints only from within the portion of the image.

4. The method of claim 3 , wherein selecting a portion of the image based on the audio recorded at the scene includes:

determining an audio direction of arrival (DOA) from the audio; and

selecting the portion of the image based on the audio DOA.

5. The method of claim 4 , wherein determining the audio DOA includes:

receiving the audio at a plurality of microphones located at the scene, whereby producing a plurality of microphone signals; and

determining the audio DOA based on the microphone signals.

6. The method of claim 1 , further comprising:

computing a plurality of local motion vectors from a video recording of the scene; and

identifying the object by comparing the local motion vectors to a database of predetermined local motion vectors corresponding to one or more objects and by comparing the keypoints to one or more keypoint signatures.

7. The method of claim 1 ,

wherein identifying the object is based on comparing the keypoints to one or more keypoint signatures.

8. The method of claim 7 , wherein the acoustic-recognition features include mel-frequency cepstral coefficients.

9. The method of claim 1 , further comprising:

determining range information for one or more objects appearing in an image; and

analyzing the keypoints based on the range information.

10. The method of claim 9 , wherein determining range information is selected from the group consisting of determining range information using an auto-focus camera, determining range information using a multi-camera image disparity estimation and any combination of the foregoing.

11. An apparatus, comprising:

an audio processor configured to compute a plurality of acoustic-recognition features from audio recorded at a scene;

a keypoint selector configured to select keypoints corresponding to an object based on a sound source type; and

a matching device configured to identify the object based on the selected keypoints and comparing the acoustic-recognition features to predetermined acoustic-recognition features corresponding to one or more objects to determine the sound source type of the object.

12. The apparatus of claim 11 , further comprising:

a keypoint detector configured to identify a plurality of keypoints in an image of a scene;

wherein the keypoint selector is configured to select one or more keypoint signatures corresponding to one or more objects, based on audio recorded at the scene; and

wherein the matching device is configured to compare the keypoints to the keypoint signatures to identify an object in the scene.

13. The apparatus of claim 11 , further comprising:

a first selector configured to select a portion of an image of the scene based on the audio recorded at the scene; and

a second selector configured to select the keypoints only from within the portion of the image.

14. The apparatus of claim 13 , wherein the first selector includes:

a detector configured to determine an audio direction of arrival (DOA) from the audio; and

a third selector configured to select the portion of the image based on the audio DOA.

15. The apparatus of claim 14 , wherein the detector includes:

a plurality of microphones located at the scene for receiving the audio, producing a plurality of microphone signals; and

an audio processor configured to determine the audio DOA based on the microphone signals.

16. The apparatus of claim 11 , further comprising:

a video processor configured to compute a plurality of local motion vectors from a video recording of the scene;

wherein the matching device is configured to identify the object by comparing the local motion vectors to a database of predetermined local motion vectors corresponding to one or more objects and by comparing the keypoints to one or more keypoint signatures.

17. The apparatus of claim 11 ,

wherein the matching device is configured to identify the object by comparing the keypoints to one or more keypoint signatures.

18. The apparatus of claim 17 , wherein the acoustic-recognition features include mel-frequency cepstral coefficients.

19. The apparatus of claim 11 , further comprising:

a range detector configured to determine range information for one or more objects appearing in an image; and

a keypoint detector configured to analyze the keypoints based on the range information.

20. The apparatus of claim 19 , wherein the range detector includes a detector selected from the group consisting of an auto-focus camera, a multi-camera array and any combination of the foregoing.

21. An apparatus, comprising:

means for computing a plurality of acoustic-recognition features from audio recorded at a scene;

means for comparing the acoustic-recognition features to predetermined acoustic-recognition features corresponding to one or more objects to determine a sound source type of an object;

means for selecting keypoints corresponding to the object based on the sound source type;

and

means for identifying the object based on the selected keypoints and the sound source type.

22. The apparatus of claim 21 , further comprising:

means for selecting one or more keypoint signatures corresponding to one or more objects, based on audio recorded at the scene;

means for identifying a plurality of keypoints in an image of the scene; and

means for comparing the keypoints to the keypoint signatures to identify the object in the scene.

23. The apparatus of claim 21 , further comprising:

means for selecting a portion of an image of the scene based on the audio recorded at the scene; and

means for selecting the keypoints only from within the portion of the image.

24. The apparatus of claim 23 , wherein the means for selecting a portion of the image based on the audio recorded at the scene includes:

means for determining an audio direction of arrival (DOA) from the audio; and

means for selecting the portion of the image based on the audio DOA.

25. The apparatus of claim 24 , wherein means for determining the audio DOA includes:

means for receiving the audio at a plurality of microphones located at the scene, whereby producing a plurality of microphone signals; and

means for determining the audio DOA based on the microphone signals.

26. The apparatus of claim 21 , further comprising:

means for computing a plurality of local motion vectors from a video recording of the scene; and

means for identifying the object by comparing the local motion vectors to a database of predetermined local motion vectors corresponding to one or more objects and by comparing the keypoints to one or more keypoint signatures.

27. The apparatus of claim 21 , further comprising

means for comparing the keypoints to one or more keypoint signatures.

28. The apparatus of claim 27 , wherein the acoustic-recognition features include mel-frequency cepstral coefficients.

29. The apparatus of claim 21 , further comprising:

means for determining range information for one or more objects appearing in an image; and

means for analyzing the keypoints based on the range information.

30. The apparatus of claim 29 , wherein means for determining range information is selected from the group consisting of means for determining range information using an auto-focus camera, means for determining range information using a multi-camera image disparity estimation and any combination of the foregoing.

31. A non-transitory tangible computer-readable medium embodying a set of instructions executable by one or more processors, comprising:

code for computing a plurality of acoustic-recognition features from audio recorded at a scene;

code for comparing the acoustic-recognition features to predetermined acoustic-recognition features corresponding to one or more objects to determine a sound source type of an object;

code for selecting keypoints corresponding to the object based on the sound source type;

and

code for identifying the object based on the selected keypoints and the sound source type.

32. The computer-readable medium of claim 31 , further comprising:

code for selecting one or more keypoint signatures corresponding to one or more objects, based on audio recorded at the scene;

code for identifying a plurality of keypoints in an image of the scene; and

code for comparing the keypoints to the keypoint signatures to identify the object in the scene.

33. The computer-readable medium of claim 31 , further comprising:

code for selecting a portion of an image based on the audio recorded at the scene; and

code for selecting the keypoints only from within the portion of the image.

34. The computer-readable medium of claim 33 , wherein the code for selecting a portion of the image based on the audio recorded at the scene includes:

code for determining an audio direction of arrival (DOA) from the audio; and

code for selecting the portion of the image based on the audio DOA.

35. The computer-readable medium of claim 34 , wherein code for determining the audio DOA includes:

code for receiving the audio at a plurality of microphones located at the scene, whereby producing a plurality of microphone signals; and

code for determining the audio DOA based on the microphone signals.

36. The computer-readable medium of claim 31 , further comprising:

code for computing a plurality of local motion vectors from a video recording of the scene; and

code for identifying the object by comparing the local motion vectors to a database of predetermined local motion vectors corresponding to one or more objects and by comparing the keypoints to one or more keypoint signatures.

37. The computer-readable medium of claim 31 , further comprising

code for comparing the keypoints to one or more keypoint signatures.

38. The computer-readable medium of claim 37 , wherein the acoustic-recognition features include mel-frequency cepstral coefficients.

39. The computer-readable medium of claim 31 , further comprising:

code for determining range information for one or more objects appearing in an image; and

code for analyzing the keypoints based on the range information.

40. The computer-readable medium of claim 39 , wherein code for determining range information is selected from the group consisting of code for determining range information using an auto-focus camera, code for determining range information using a multi-camera image disparity estimation and any combination of the foregoing.