Multimodal state circulation

1. A method comprising:

receiving a request from a user device to perform a task;

obtaining a multimodal dialog corresponding to the task, wherein the multimodal dialog includes questions that can be responded to with a combination of voice and non-voice inputs;

providing a sequence of ordered protocol buffers associated with the multimodal dialog to the user device, wherein each protocol buffer includes information about a question of the multimodal dialog;

receiving a bundle of two or more annotated protocol buffers from the user device representing a local state of the multimodal dialog on the user device, the bundle of two or more annotated protocol buffers including a voice annotated protocol buffer identifying a voice input response to a question of the multimodal dialog and one or more non-voice annotated protocol buffers identifying non-voice responses to corresponding questions of the multimodal dialog that preceded the question of the multimodal dialog resulting in the voice input response; and

using the bundle of two or more annotated protocol buffers to update a state of the multimodal dialog and to interpret the voice annotated protocol buffer.

2. The method of claim 1 , wherein each protocol buffer is a DialogTurnIntent (DTI).

3. The method of claim 1 , comprising: providing one or more additional protocol buffers to the user device in response to updating the state of the multimodal dialog following the voice input.

4. The method of claim 1 , wherein the sequence of the ordered protocol buffers encompass the entire multimodal dialog for the task.

5. The method of claim 1 , comprising: completing the task once values for the multimodal dialog are determined based on user inputs responsive to questions provided by the sequence of the ordered protocol buffers.

6. The method of claim 5 , wherein completing the task includes providing a calendar item generated using the values of the multimodal dialog.

7. The method of claim 1 , wherein the multimodal dialog indicates particular values needed to complete the task and the state of the multimodal dialog identifies a current position in the multimodal dialog.

8. A method comprising:

receiving a user input at a user device to perform a task;

providing the user input to a dialog system to generate a dialog;

receiving multiple protocol buffers for the dialog, wherein each protocol buffer includes information about a question of the dialog;

presenting a first prompt for a first protocol buffer to a user;

receiving a non-verbal response to the first prompt;

updating a local state of the dialog at the user device with the non-verbal response including annotating the first protocol buffer identifying the non-voice response and presenting a second prompt for a second protocol buffer to the user;

receiving a voice input in response to the second prompt; and

in response to receiving the voice input, updating the local state of the dialog with the verbal response including annotating the second protocol buffer identifying the voice response and providing the voice input and a bundle of both the first and second annotated protocol buffers to the dialog system.

9. The method of claim 8 , wherein the multiple protocol buffers are received as part of a resource set that indicates an order of alternative protocol buffers.

10. The method of claim 8 , wherein presenting the first prompt for the first protocol buffer includes providing a user interface associated with the first prompt to which the user can input the non-verbal response.

11. The method of claim 8 , wherein updating the local state of the dialog includes annotating the corresponding protocol buffer with the received non-verbal response.

12. The method of claim 8 , wherein the user input to perform a task is a user input to generate a calendar item.

13. The method of claim 8 , further comprising:

receiving a calendar item that includes values populated using the received voice input and non-verbal response.

14. The method of claim 8 , further comprising:

receiving one or more additional protocol buffers for the dialog from the dialog system; and

presenting a first prompt for a first additional protocol buffer to the user.

15. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

receiving a request from a user device to perform a task;

obtaining a multimodal dialog corresponding to the task, wherein the multimodal dialog includes questions that can be responded to with a combination of voice and non-voice inputs;

providing a sequence of ordered protocol buffers associated with the multimodal dialog to the user device, wherein each protocol buffer includes information about a question of the multimodal dialog;

receiving a bundle of two or more annotated protocol buffers from the user device representing a local state of the multimodal dialog on the user device, the bundle of two or more annotated protocol buffers including a voice annotated protocol buffer identifying a voice input response to a question of the multimodal dialog and one or more non-voice annotated protocol buffers identifying non-voice responses to corresponding questions of the multimodal dialog that preceded the question of the multimodal dialog resulting in the voice input response; and

using the bundle of two or more annotated protocol buffers to update a state of the multimodal dialog and to interpret the voice annotated protocol buffer.

16. The system of claim 15 , wherein the instructions are further operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: completing the task once values for the multimodal dialog are determined based on user inputs responsive to questions provided by the sequence of the ordered protocol buffers.

17. The system of claim 15 , wherein the multimodal dialog indicates particular values needed to complete the task and the state of the multimodal dialog identifies a current position in the multimodal dialog.

18. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

receiving a user input at a user device to perform a task;

providing the user input to a dialog system to generate a dialog;

receiving multiple protocol buffers for the dialog, wherein each protocol buffer includes information about a question of the dialog;

presenting a first prompt for a first protocol buffer to a user;

receiving a non-verbal response to the first prompt;

updating a state of the dialog with the non-verbal response including annotating the first protocol buffer identifying the non-voice response and presenting a second prompt for a second protocol buffer to the user;

receiving a voice input in response to the second prompt; and

in response to receiving the voice input, updating the local state of the dialog with the verbal response including annotating the second protocol buffer identifying the voice response and providing the voice input and a bundle of both the first and second annotated protocol buffers to the dialog system.

19. The system of claim 18 , wherein the multiple protocol buffers are received as part of a resource set that indicates an order of alternative protocol buffers.

20. The system of claim 18 , wherein presenting the first prompt for the first protocol buffer includes providing a user interface associated with the first prompt to which the user can input the non-verbal response.