Method and apparatus for a verbo-manual gesture interface
An interface system including a manipulandum adapted to be moveable according to a manual gesture imparted by the user; a sensor adapted to detect a characteristic of the manual gesture imparted to the manipulandum and to generate a sensor signal representing the detected characteristic of the manual gesture; a microphone adapted to detect a characteristic of an utterance spoken by the user and to generate an audio signal representing the detected characteristic of the spoken utterance; and a control system adapted receive the generated sensor and audio signals and to transmit a command signal to an electronic device via a communication link, the command signal being based on the generated sensor and audio signals and the time synchronization between them.
1. A verbo-manual gesture interface system, comprising:
a manipulandum adapted to be movable according to a multi-stroke manual gesture imparted by a user, the multi-stroke manual gesture comprising a plurality of characteristic manual strokes;
a sensor adapted to detect each of the plurality of characteristic manual strokes within the multi-stroke manual gesture imparted to the manipulandum and to generate a sensor signal representing the timing of each of the plurality of characteristic manual strokes within the multi stroke manual gesture;
a microphone adapted to detect a spoken phrase uttered by the user, the spoken phrase including a plurality of uttered words spoken by the user and to generate an utterance signal representing the timing of each of the plurality of uttered words within the spoken phrase; and
a control system adapted to receive the generated sensor signal and the generated utterance signal and to transmit a command signal to at least one electronic device, the command signal being dependent, at least in part, upon a predetermined temporal relationship between a particular one of the plurality of characteristic manual strokes within the multi-stroke manual gesture and a particular one of the plurality of uttered words within the spoken phrase,
wherein the predetermined temporal relationship requires that the particular one of the plurality of characteristic manual strokes within the multi-stroke manual gesture is imparted with substantial simultaneity with the particular one of the plurality of uttered word within the spoken phrase.
2. The system of claim 1 , wherein the control system transmits the command signal on further dependence upon a second predetermined temporal relationship between a second one of the plurality of characteristic manual strokes within the multi-stroke manual gesture and a second one of the plurality of uttered words within the spoken phrase.
3. The system of claim 2 , wherein the predetermined temporal relationship requires that the second one of the plurality of characteristic manual strokes within the multi-stroke manual gesture is imparted with substantial simultaneity with the second one of the plurality of uttered word within the spoken phrase.
4. The system of claim 1 , wherein the sensor comprises an accelerometer.
5. A verbo-manual interface method, comprising the steps of:
detecting each of a plurality of characteristic manual strokes within a multi-stroke manual gesture imparted to a manipulandum by a user;
generating a sensor signal representing the timing of each of the plurality of characteristic manual strokes within the multi-stroke manual gesture;
detecting a spoken phrase uttered by a user, the spoken phrase including a plurality of uttered words spoken by the user;
generating an utterance signal representing the timing of each of the plurality of uttered words within the spoken phrase; and
controlling at least one electronic device in dependence upon a predetermined temporal relationship between a particular one of the plurality of characteristic manual strokes within the multi-stroke manual gesture and a particular one of the plurality of uttered words within the spoken phrase,
wherein the controlling step comprises the predetermined temporal relationship requiring that the first manual stroke is imparted with substantial simultaneity with the first spoken word and that the second manual stroke is imparted with a substantial simultaneity with the second spoken word.
6. The method of claim 5 ,
wherein the manual stroke detecting step comprises detecting the multi-stroke manual gesture as at least a first manual stroke and a second manual stroke, and
wherein the spoken phrase detecting step comprises detecting the plurality of uttered words as at least a first spoken word and a second spoken word.
7. The method of claim 5 , wherein the manual stroke detecting step comprises detecting at least one of the plurality of characteristic manual strokes as one of a circular clockwise stroke and a circular counter-clockwise stroke.
8. The method of claim 5 , wherein the manual stroke detecting step comprises detecting at least one of the characteristic manual strokes as one of a forward thrust stroke and a backwards pull stroke.
9. The method of claim 5 , wherein the manual stroke detecting step comprises detecting at least one of the characteristic manual strokes by an accelerometer.
10. The method of claim 5 , wherein the controlling step comprises controlling the at least one electronic device as a function of a magnitude value generated for at least one of the plurality of characteristic manual strokes.
11. The method of claim 5 , wherein the controlling step comprises controlling the at least one electronic device as a function of a volume value generated for at least one of the plurality of uttered words.
12. The method of claim 5 ,
wherein the manual stroke detecting step comprises detecting the multi-stroke manual gesture as at least a first manual stroke and a second manual stroke,
wherein the spoken phrase detecting step comprises detecting the plurality of uttered words as at least a first spoken word and a second spoken word,
wherein the manual stroke detecting step comprises detecting at least one of the plurality of characteristic manual strokes as one of a circular clockwise stroke and a circular counter clockwise stroke,
wherein the manual stroke detecting step comprises detecting at least one of the characteristic manual strokes as one of a forward thrust stroke and a backwards pull stroke, and
wherein the manual stroke detecting step comprises detecting at least one of the characteristic manual strokes by an accelerometer.