Systems and methods for extracting meaning from multimodal inputs using finite-state devices
View Patent ↗Multimodal utterances contain a number of different modes. These modes can include speech, gestures, and pen, haptic, and gaze inputs, and the like. This invention use recognition results from one or more of these modes to provide compensation to the recognition process of one or more other ones of these modes. In various exemplary embodiments, a multimodal recognition system inputs one or more recognition lattices from one or more of these modes, and generates one or more models to be used by one or more mode recognizers to recognize the one or more other modes. In one exemplary embodiment, a gesture recognizer inputs a gesture input and outputs a gesture recognition lattice to a multimodal parser. The multimodal parser generates a language model and outputs it to an automatic speech recognition system, which uses the received language model to recognize the speech input that corresponds to the recognized gesture input.
1. A multimodal recognition system that inputs an utterance comprising a plurality of associated modes, the system comprising:
a plurality of mode recognition systems, each mode recognition system usable to recognize ones of the associated modes, each mode recognition system outputting a recognition result for each associated mode; and
a multimodal recognition system that inputs recognition results from at least a first one of the plurality of mode recognition systems, that generates, for at least a second one of the plurality of mode recognition systems distinct from the at least first one of the plurality of mode recognition systems, at least one recognition model based on the recognition results from the at least first one of the plurality of mode recognition systems, and that outputs each generated recognition model to a corresponding one of the at least second one of the plurality of mode recognition systems;
wherein each corresponding mode recognition system of the at least second one of the plurality of mode recognition systems generates the recognition result for the associated mode for that corresponding mode recognition system based on the corresponding generated recognition model.
2. The multimodal recognition system of claim 1 , wherein the first one of the plurality of mode recognition systems inputs at least one first mode and outputs a first mode recognition lattice as the recognition result to the multimodal recognition system.
3. The multimodal recognition system of claim 2 , wherein the multimodal recognition system includes a first subsystem that inputs the first mode recognition lattice and a first finite-state transducer that relates a first one of the different modes to a second one of the different modes, the first subsystem outputting a second finite-state transducer based on the first mode recognition lattice and the first finite-state transducer.
4. The multimodal recognition system of claim 3 , wherein the multimodal recognition system further comprises a second subsystem that inputs the second finite-state transducer and that outputs a projection of the second finite-state transducer as the recognition model to the second one of the plurality of mode recognition systems.
5. The multimodal recognition system of claim 4 , wherein the second one of the plurality of mode recognition systems inputs the second mode and outputs a second mode recognition result based on the recognition model output by the multimodal recognition system.
6. The multimodal recognition system of claim 1 , wherein the first one of the plurality of mode recognition systems comprises a gesture recognition system, and the second one of the plurality of mode recognition systems comprises a speech recognition system.
7. The multimodal recognition system of claim 6 , wherein the gesture recognition system inputs at least one gesture mode and outputs a gesture recognition lattice as the recognition result to the multimodal recognition system.
8. The multimodal recognition system of claim 7 , wherein the multimodal recognition system includes a first subsystem that inputs the gesture recognition lattice and a first finite-state transducer that relates gesture utterances to speech utterances, the first subsystem outputting a gesture/speech recognition model finite-state transducer based on the gesture recognition lattice and the first finite-state transducer.
9. The multimodal recognition system of claim 8 , wherein the multimodal recognition system further comprises a projection subsystem that inputs the gesture/speech recognition model finite-state transducer and that outputs a projection of the gesture/speech recognition model finite-state transducer as a speech recognition model to the speech recognition system.
10. The multimodal recognition system of claim 9 , wherein the speech recognition system inputs a speech mode and outputs a speech recognition result based on the speech recognition model output by the multimodal recognition system.
11. The multimodal recognition system of claim 10 , wherein the speech recognition system comprises:
a speech processing subsystem that inputs a speech signal and outputs a feature vector lattice;
a phonetic recognition subsystem that inputs the feature vector lattice and an acoustic model lattice and outputs a phone lattice;
a word recognition subsystem that inputs the phone lattice and a lexicon lattice and outputs a word lattice; and
a spoken mode recognition subsystem that inputs the word lattice and the speech recognition model from the multimodal recognition subsystem and outputs a recognized mode as the speech recognition result.
12. The multimodal recognition system of claim 9 , wherein the speech recognition model is a speech recognition lattice.
13. The multimodal recognition system of claim 12 , wherein the speech recognition lattice is one of a grammar model lattice and a language model lattice.
14. The multimodal recognition system of claim 7 , wherein the gesture recognition system comprises a gesture feature recognition subsystem that inputs a gesture mode and outputs a gesture feature lattice and a gesture recognition subsystem that inputs the gesture feature lattice and outputs the gesture recognition lattice.
15. The multimodal recognition system of claim 1 , further comprising a plurality of mode input devices, at least two of the plurality of mode input devices capable of inputting different modes.
16. The multimodal recognition system of claim 15 , wherein the plurality of mode input devices comprise at least two of a gesture input device, a speech input device, a pen input device, a computer vision device, a haptic input device, a gaze input device, and a body motion input device.
17. The multimodal recognition system of claim 15 , wherein at least two of the plurality of input devices are combined into a single multimodal input device.
18. The multimodal recognition system of claim 1 , wherein the plurality of mode recognition systems comprise at least two of a gesture recognition subsystem, a speech recognition subsystem, a pen input recognition subsystem, a computer vision recognition subsystem, a haptic recognition subsystem, a gaze recognition subsystem, and a body motion recognition system.
19. A method for recognizing a multimodal utterance comprising a plurality of associated modes, the method comprising:
inputting at least a first mode and a second mode that is different from the first mode;
generating a first mode recognition lattice based on the first mode;
composing the first mode recognition lattice with a first finite-state transducer that relates the first mode to the second mode to generate a second finite-state transducer;
generating a projection of the second finite-state transducer; and
recognizing the second mode using the projection as a recognition model usable in recognizing the second mode.
20. The method of claim 19 , wherein generating the first mode recognition lattice based on the first mode comprises:
extracting a plurality of first mode features from the first mode; and
generating the first mode recognition lattice from the extracted features.
21. The method of claim 19 , wherein the first mode is a gesture mode and the second mode is a speech mode.
22. The method of claim 21 , wherein generating the first mode recognition lattice based on the first mode comprises generating a gesture recognition lattice based on the gesture mode.
23. The method of claim 22 , wherein generating the gesture recognition lattice based on the gesture mode comprises extracting gesture features from the gesture mode.
24. The method of claim 23 , wherein generating the gesture recognition lattice based on the gesture mode further comprises generating the gesture recognition lattice based on the extracted gesture features.
25. The method of claim 21 , further comprising:
converting the speech mode into a speech signal;
generating a feature vector lattice based on the speech signal;
generating a phone lattice based on the feature vector lattice and an acoustic model lattice; and
generating a word lattice based on the phone lattice and a lexicon lattice.
26. The method of claim 25 , wherein recognizing the second mode using the projection as a recognition model comprises using the projection as a speech recognition model.
27. The method of claim 26 , wherein recognizing the second mode using the projection as the speech recognition model comprises outputting a recognized speech mode as the speech recognition result based on the word lattice and the speech recognition model.
28. The method of claim 26 , wherein the speech recognition model is a speech recognition lattice.
29. The method of claim 28 , wherein the speech recognition lattice is one of a grammar model lattice and a language model lattice.
30. The method of claim 19 , wherein the first mode is one of a gesture mode, a speech mode, a pen input mode, a computer vision input mode, a haptic mode, a gaze mode, and a body motion mode.
31. The method of claim 30 , wherein the second mode is a different one of the gesture mode, the speech mode, the pen input mode, the computer vision input mode, the haptic mode, the gaze mode, and the body motion mode.