Method and system for assisting users in interacting with multi-modal dialog systems

1. A method for assisting a user in interacting with a multi-modal dialog apparatus, the method comprising:

interpreting by a processor of the multi-modal dialog apparatus a What Can I Do (WCID) type of question from the user that is received at a user interface device coupled to the multi-modal dialog apparatus;

generating by the processor one or more user multi-modal utterances as sequences of words that a user may provide in the multi-modal inputs in a next turn of the dialog based on the WCID question and a multi-modal grammar, wherein the multi-modal grammar is based on a current context of a multi-modal dialog; and

conveying the one or more user multi-modal utterances to the user at a user interface device coupled to the multi-modal dialog apparatus.

2. The method according to claim 1 further comprising de-referencing the one or more user multi-modal utterances using a visual context of the multi-modal dialog.

3. The method according to claim 2 further comprising storing the visual context of the multi-modal dialog.

4. The method according to claim 1 , wherein generating the one or more user multi-modal utterances comprises generating the multi-modal grammar based on the current context of the multi-modal dialog.

5. The method according to claim 1 further comprising ranking the one or more user multi-modal utterances based on one or more of a group consisting of the current context of the multi-modal dialog, a user model and modality availability.

6. A multi-modal dialog apparatus comprising:

a user interface device;

a processor; and

a memory, wherein the memory stores programming instructions organized into functional groups that control the processor, the functional groups comprising

a multi-modal input fusion (MMIF) component, the multi-modal input fusion component accepting a What Can I Do (WCID) type of question from the user through the user interface device;

a dialog manager, the dialog manager generating a multi-modal grammar based on a current context of a multi-modal dialog;

a multi-modal utterance generator, the multi-modal utterance generator generating one or more user multi-modal utterances through the user interface device, as sequences of words that a user may provide in the multi-modal inputs in a next turn of the dialog based on the question and the multi-modal grammar.

7. The multi-modal dialog apparatus of claim 6 wherein the functional groups further comprise a multi-modal utterance ranker, the multi-modal utterance ranker ranking the one or more user multi-modal utterances based on one or more of a group consisting of a current context of the multi-modal dialog, a user model and modality availability.

8. The multi-modal dialog apparatus of claim 6 wherein the functional groups further comprise a visual context manager, the visual context manager storing a visual context of the multi-modal dialog.

9. The multi-modal dialog apparatus of claim 8 , wherein the multi-modal utterance generator generates the one or more user multi-modal utterances based on the visual context of the multi-modal dialog.

10. The multi-modal dialog apparatus of claim 6 wherein the functional groups further comprise an output generator, the output generator conveying the one or more user multi-modal utterances to the user.

11. The multi-modal dialog apparatus of claim 6 , wherein the MMIF component maintains and updates the modality availability.

12. The multi-modal dialog apparatus of claim 6 , wherein the dialog manager interprets the question from the user.

13. The multi-modal dialog apparatus of claim 6 , wherein the dialog manager maintains and updates the current context of the dialog.