Object detection and tracking with audio and optical signals
View Patent ↗The technology disclosed addresses these problems by supplementing optical gesture recognition with the ability to recognize touch gestures. This capability allows the user to execute intuitive gestures involving contact with a surface. For example, in low-light situations where free-form gestures cannot be recognized optically with a sufficient degree of reliability, a device implementing the invention may switch to a touch mode in which touch gestures are recognized. In implementations, two contact microphones or other vibratory or acoustical sensors, that are coupled to an optical motion-capture and are in contact with a surface that a user touches, are monitored. When the contact microphones detect audio signals (or other vibrational phenomena) generated by contact of an object with the surface that the user touches, a position of the object traveling across and in contact with the surface is tracked.
1. A method of operating an optical motion-capture system in low-light situations, the method including:
monitoring at least two contact microphones coupled to the optical motion-capture system, wherein the contact microphones are in contact with a surface; and
in response to detection of audio signals generated by contact of an object with the surface determining a surface location by:
measuring a time difference of arrival (TDOA) of the audio signals at the contact microphones and mapping a curve, including a plurality of possible surface locations of the contact, using the measured time difference of arrival;
analyzing at least one image, captured by a camera of the optical motion-capture system, of the object in contact with the surface; and
using the image analysis, selecting one of the plurality of possible surface locations included in the mapped curve, as the surface location of the contact;
repeatedly measuring time difference of arrival of the audio signals at the contact microphones responsive to successive contacts of the object with the surface; and
analyzing a sequence of images of the successive contacts of the object with the surface; thereby calculating a path of the object traveling across and in contact with the surface.
2. The method of claim 1 , further including comparing the path to a plurality of path templates and identifying a template that best matches the path.
3. The method of claim 1 , further including constructing a 3D model of the object and tracking movement thereof over the surface based on the sequence of images and the audio signals.
4. The method of claim 1 , further including recognizing at least one of an image sequence or an audio signal that indicates a trajectory of the object.
5. The method of claim 1 , further including recognizing at least one of an image sequence or an audio signal that indicates a gesture of the object.
6. A method of tracking a path of an object of interest traveling across and in contact with a surface, the method comprising:
capturing, with a camera, a sequence of images of a field of view that includes at least a portion of the surface;
capturing audio signals through the surface using at least two microphones;
computationally constructing a 3D model of the object and tracking movement of the object over the surface based on the sequence of images and the audio signals, including;
measuring a time difference of arrival (TDOA) of the audio signals at the microphones and mapping a curve, including a plurality of possible surface locations of the object, using the captured audio signals;
analyzing at least one of the sequence of images; and
using the image analysis, selecting one of the plurality of possible surface locations, included in the mapped curve, as a position of the object; and
repeatedly measuring time difference of arrival of the audio signals at the microphones and performing the mapping, the analyzing and the selecting responsive to additional contacts of the object with the surface, thereby determining a path for the object traveling across the surface.
7. The method of claim 6 , wherein the 3D model includes a shape of the object.
8. The method of claim 6 , wherein tracking movement includes calculating a direction of the object by computationally analyzing the audio signals.
9. The method of claim 8 , wherein the position and direction of the object are calculated using a time difference of arrival of the audio signals at the microphones.
10. The method of claim 6 , further including filtering the audio signals to exclude frequencies outside a defined range.
11. The method of claim 6 , further including recognizing at least one of an image sequence or an audio signal that indicates a trajectory or a gesture of the tracked object.
12. The method of claim 11 , wherein the trajectory or the gesture dictates a change in operational mode.
13. An optical motion-capture system, including:
a camera for capturing images of an object within a field of view thereof;
two contact microphones coupled to the camera and in contact with a surface; and
a sensor for monitoring the contact microphones;
a control module for tracking a position of the object, in response to detection of audio signals generated by contact of the object with the surface, by:
measuring a time difference of arrival (TDOA) of the audio signals at the contact microphones and mapping a curve, including a plurality of possible surface locations of the contact, using the measured time difference of arrival;
analyzing at least one image, captured by the camera of the optical motion-capture system, of the object in contact with the surface; and
using the image analysis, selecting one of the plurality of possible surface locations included in the mapped curve, as the surface location of the contact;
repeatedly measuring time difference of arrival of the audio signals at the contact microphones responsive to successive contacts of the object with the surface; and
analyzing a sequence of images of the successive contacts of the object with the surface; thereby calculating a path of the object traveling across and in contact with the surface.
14. The system of claim 13 , further configured to construct a 3D model of the object and tracking movement thereof over the surface based on the sequence of images and the audio signals.
15. The system of claim 13 , further configured to recognize at least one of an image sequence or an audio signal that indicates a trajectory of the tracked object.
16. The system of claim 13 , further configured to recognize at least one of an image sequence or an audio signal that indicates a gesture of the tracked object.