Multimodal responses
Systems, methods, and apparatus for using a multimodal response in the dynamic generation of client device output that is tailored to a current modality of a client device is disclosed herein. Multimodal client devices can engage in a variety of interactions across the multimodal spectrum including voice only interactions, voice forward interactions, multimodal interactions, visual forward interactions, visual only interactions etc. A multimodal response can include a core message to be rendered for all interaction types as well as one or more modality dependent components to provide a user with additional information.
1. A method implemented by one or more processors, the method comprising:
determining a client device action based on one or more instances of user interface input provided by a user of a multimodal client device;
processing, using a machine learning model, (1) sensor data from one or more sensors of the multimodal client device and (2) a multimodal response, to generate client device output,
wherein the sensor data from the one or more sensors of the multimodal client device indicate a current client device modality of the multimodal client device,
wherein the current client device modality is one of a plurality of discrete client device modalities available for the multimodal client device,
wherein the sensor data is in addition to any sensor data generated by the one or more instances of user interface input,
wherein the multimodal response includes components of output for the client device action for the plurality of discrete client device modalities,
wherein the multimodal response is received by the multimodal client device from a remote server,
wherein generating the client device output is done by the multimodal client device, and
wherein the components of output include at least (a) a core message component representing information to render for each of the plurality of discrete client device modalities and (b) one or more modality dependent components each representing corresponding information to render for one or more of the plurality of discrete client device modalities;
causing the client device output to be rendered by one or more user interface output devices of the multimodal client device;
while at least part of the client device output is being rendered by the one or more user interface output devices of the multimodal client device:
detecting a switch of the multimodal client device, based on alternative sensor data from the one or more sensors of the multimodal client device, from the current client device modality to a discrete new client device modality;
in response to detecting the switch, generating alternative client device output based on processing (1) the alternative sensor data from the one or more sensors of the multimodal client device and (2) the multimodal response, using the machine learning model; and
causing the alternative client device output to be rendered by the one or more user interface output devices of the multimodal client device.
2. The method of claim 1 , wherein the multimodal response is received by the multimodal client device from the remote server in response to a request, transmitted to the remote server by the client device, that is based on the user interface input, and wherein determining the current client device modality of the multimodal client device is by the multimodal client device and occurs after transmission of the request.
3. The method of claim 1 ,
wherein the client device output includes audible output rendered via at least one speaker of the one or more user interface output devices of the multimodal client device and visual output rendered via at least one display of the one or more user interface output devices,
wherein the alternative client device output lacks the visual output, and
wherein causing the alternative client device output to be rendered by the multimodal client device comprises ceasing rendering of the visual output by the at least one display.
4. The method of claim 1 , wherein the current client device modality is a voice only interaction and the client device output is rendered via one or more speakers of the one or more user interface output devices.
5. The method of claim 1 , wherein the current client device modality is a voice forward interaction, the core message component of the client device output is rendered via only one or more speakers of the one or more user interface output devices, and the one or more modality dependent components of the client device output are rendered via a touch screen of the one or more user interface output devices.
6. The method of claim 1 , wherein the current client device modality is a multimodal interaction, the client device output is rendered via one or more speakers and via a touch screen of the one or more user interface output devices.
7. The method of claim 1 , wherein the current device modality is a visual forward interaction, the core message component of the client device output is rendered via only a touch screen of the one or more user interface output devices, and the one or more modality dependent components of the client device output are rendered via one or more speakers of the one or more user interface output devices.
8. The method of claim 1 , wherein the current device modality is a visual only interaction, and the client device output is rendered via only a touch screen of the one or more user interface output devices.
9. A multimodal client device, comprising:
one or more microphones;
one or more sensors that are in addition to the one or more microphones;
one or more speakers;
one or more displays;
memory storing instructions;
one or more processors executing the instructions to perform the method of:
determining a client device action based on one or more instances of user interface input provided by a user of the multimodal client device;
processing, using a machine learning model, (1) sensor data from the one or more sensors of the multimodal client device and (2) a multimodal response, to generate client device output,
wherein the sensor data from the one or more sensors of the multimodal client device indicate a current client device modality of the multimodal client device,
wherein the current client device modality is one of a plurality of discrete client device modalities available for the multimodal client device,
wherein the sensor data is in addition to any sensor data generated by the one or more instances of user interface input,
wherein the multimodal response includes components of output for the client device action for the plurality of discrete client device modalities,
wherein the multimodal response is received by the multimodal client device from a remote server,
wherein generating the client device output is done by the multimodal client device, and
wherein the components of output include at least (a) a core message component representing information to render for each of the plurality of discrete client device modalities and (b) one or more modality dependent components each representing corresponding information to render for one or more of the plurality of discrete client device modalities;
causing the client device output to be rendered by one or more user interface output devices of the multimodal client device;
while at least part of the client device output is being rendered by the one or more user interface output devices of the multimodal client device:
detecting a switch of the multimodal client device, based on alternative sensor data from the one or more sensors of the multimodal client device, from the current client device modality to a discrete new client device modality;
in response to detecting the switch, generating alternative client device output based on processing (1) the alternative sensor data from the one or more sensors of the multimodal client device and (2) the multimodal response, using the machine learning model; and
causing the alternative client device output to be rendered by the one or more user interface output devices of the multimodal client device.
10. The multimodal client device of claim 9 , wherein the multimodal response is received by the multimodal client device from the remote server in response to a request, transmitted to the remote server by the client device, that is based on the user interface input, and wherein determining the current client device modality of the multimodal client device is by the multimodal client device and occurs after transmission of the request.
11. The multimodal client device of claim 9 ,
wherein the client device output includes audible output rendered via at least one speaker of the one or more user interface output devices of the multimodal client device and visual output rendered via at least one display of the one or more user interface output devices,
wherein the alternative client device output lacks the visual output, and
wherein causing the alternative client device output to be rendered by the multimodal client device comprises ceasing rendering of the visual output by the at least one display.
12. The multimodal client device of claim 9 , wherein the current client device modality is a voice only interaction and the client device output is rendered via one or more speakers of the one or more user interface output devices.
13. The multimodal client device of claim 9 , wherein the current client device modality is a voice forward interaction, the core message component of the client device output is rendered via only one or more speakers of the one or more user interface output devices, and the one or more modality dependent components of the client device output are rendered via a touch screen of the one or more user interface output devices.
14. The multimodal client device of claim 9 , wherein the current client device modality is a multimodal interaction, the client device output is rendered via one or more speakers and via a touch screen of the one or more user interface output devices.
15. The multimodal client device of claim 9 , wherein the current device modality is a visual forward interaction, the core message component of the client device output is rendered via only a touch screen of the one or more user interface output devices, and the one or more modality dependent components of the client device output are rendered via one or more speakers of the one or more user interface output devices.
16. The multimodal client device of claim 9 , wherein the current device modality is a visual only interaction, and the client device output is rendered via only a touch screen of the one or more user interface output devices.