Apparatus and method for predicting user's intention based on multimodal information
Disclosed are an apparatus and method of deducing a user's intention using multimodal information. The user's intention deduction apparatus includes a first predictor to predict a part of a user's intention using at least one piece of motion information, and a second predictor to predict the user's intention using the predicted part of the user's intention and multimodal information received from at least one multimodal sensor.
1. A user's intention deduction apparatus, comprising:
a detector configured to generate motion information based upon detected motion of a user;
a first predictor configured to predict a part of the user's intention using the motion information; and
a second predictor configured to predict the user's intention using the part of the user's intention predicted by the first predictor and multimodal information received from at least one multimodal sensor, wherein
the first predictor predicts the part of the user's intention to be the selection of an object displayed by a screen based upon an angle and a distance between the user's two hands.
2. The user's intention deduction apparatus of claim 1 , wherein the first predictor is further configured to generate a control signal for executing an operation to be performed when predicting the user's intention using the predicted part of the user's intention.
3. The user's intention deduction apparatus of claim 2 , wherein the control signal is configured to control operation of the multimodal sensor which is controlled by the user's intention deduction apparatus.
4. The user's intention deduction apparatus of claim 1 , wherein the second predictor is further configured to interpret the multimodal information received from the multimodal sensor in association with the predicted part of the user's intention to predict the user's intention.
5. The user's intention deduction apparatus of claim 4 , wherein, in response to the predicted part of the user's intention being an intention of selecting an object displayed on a display screen and voice being input from the multimodal sensor, the second predictor is further configured to interpret the voice in association with the intention of selecting the object to predict the user's intention.
6. The user's intention deduction apparatus of claim 1 , wherein the second predictor is further configured to predict the user's intention using the multimodal information received from the at least one multimodal sensor, within a range of the predicted part of the user's intention.
7. The user's intention deduction apparatus of claim 6 , wherein, in response to the predicted part of the user's intention being an operation of moving a microphone close to the user's mouth, the second predictor is further configured to:
sense a sound signal; and
extract and analyze characteristics of the sound signal to predict the user's intention.
8. The user's intention deduction apparatus of claim 7 , wherein the second predictor is further configured to:
detect a voice section from the sound signal; and
predict, in response to detecting the voice section from the sound signal, the user's intention as a voice instruction input.
9. The user's intention deduction apparatus of claim 8 , wherein the second predictor is further configured to predict, in response to detecting breath sound from the sound signal, the user's intention as a blow.
10. The user's intention deduction apparatus of claim 1 , wherein, in response to the predicted part of the user's intention being to select an object displayed on a display screen, the second predictor is further configured to predict the user's intention as at least one of deletion, classification, and arrangement of the selected object using the multimodal information.
11. The user's intention deduction apparatus of claim 1 , further comprising a user's intention applying unit configured to control software or hardware that is under control of the user's intention deduction apparatus using the prediction of the user's intention.
12. A user's intention deduction method, comprising:
receiving motion information;
predicting a part of a user's intention using the received motion information;
receiving multimodal information from at least one multimodal sensor; and
predicting the user's intention using the predicted part of the user's intention and the multimodal information, wherein
the part of the user's intention is predicted to be the selection of an object displayed by a screen based upon an angle and a distance between the user's two hands.
13. The user's intention deduction method of claim 12 , further comprising generating a control signal for executing an operation to be performed when predicting the user's intention using the predicted part of the user's intention.
14. The user's intention deduction method of claim 13 , wherein the control signal controls the operation of the multimodal sensor.
15. The user's intention deduction method of claim 12 , wherein the predicting of the user's intention comprises interpreting the multimodal information received from the multimodal sensor in association with the predicted part of the user's intention.
16. The user's intention deduction method of claim 12 , wherein the user's intention is predicted using the multimodal information received from the at least one multimodal sensor, within a range of the predicted part of the user's intention.
17. The user's intention deduction method of claim 12 , further comprising controlling software or hardware using the prediction of the user's intention.
18. The apparatus of claim 11 , wherein the first predictor further predicts the part of the user's intention to be the selection of the object displayed by the screen based upon an angle and a distance between the user's face and two hands.
19. The method of claim 12 , wherein the part of the user's intention is further predicted to be the selection of the object displayed by the screen based upon an angle and a distance between the user's face and two hands.