Electronic device for processing multi-modal input, method for processing multi-modal input and server for processing multi-modal input
An electronic device is provided. The electronic device includes a housing, a touchscreen display exposed through a first portion of the housing, a microphone disposed at a second portion of the housing, a speaker disposed at a third portion of the housing, a memory disposed inside the housing, a processor disposed inside the housing, and electrically connected to the display, the microphone, the speaker, and the memory. The memory is configured to store a plurality of application programs, each of which includes a graphic user interface (GUI).
1. A portable communication device comprising:
a memory to store an image browsing application and a plurality of messenger applications;
a touchscreen display;
a microphone;
a communication circuitry to establish a wireless communication connection between a server and the portable communication device; and
a processor configured to:
receive, via the microphone, a first user input requesting to transmit at least one image stored in the memory, from a user of the portable communication device,
transmit information corresponding to the first user input to the server using the communication circuitry,
receive, from the server communicatively coupled via the wireless communication connection, a sequence of a plurality of operations of the portable communication device based on the first user input, the sequence including a first subsequence corresponding to a first set of operations and a second subsequence corresponding to a second set of operations, the first set of operations including a first operation which is performable via the image browsing application, and the second set of operations including a second operation which is performable via the plurality of messenger applications,
perform the first set of operations in the first subsequence by executing the image browsing application, the performing of the first set of operations including displaying a first user interface indicating the at least one image to be transmitted in response to receiving the sequence of the plurality of operations,
at least temporarily while at least part of the first set of operations is performed, display, via the touchscreen display, a second user interface requesting a second user input to select a messenger application among the plurality of messenger applications,
receive, via the touchscreen display or the microphone, the second user input selecting the messenger application among the plurality of messenger applications from the user,
determine, based on the second user input, one of the plurality of messenger applications to perform the second set of operations, and
display a third user interface and transmit the at least one image using the selected messenger application by performing the second set of operations in the second subsequence via the determined one of the plurality of messenger applications, the performing of the second set of operations including entering the at least one image in an input interface of the third user interface.
2. The portable communication device of claim 1 , wherein the processor is further configured to:
identify, using the server, at least one parameter based at least in part on the first user input.
3. The portable communication device of claim 1 , wherein the processor is further configured to:
execute the determined one of the plurality of messenger applications subsequently or at least partially concurrently with the image browsing application being executed.
4. The portable communication device of claim 1 , wherein the processor is further configured to:
perform at least part of the second set of operations, while execution of the first image browsing application is maintained.
5. The portable communication device of claim 1 , wherein the processor is further configured to:
transmit, using the communication circuitry, first data corresponding to the first user input to the server;
receive, using the communication circuitry, second data in response to the first data from the server; and
identify the sequence of the plurality of operations based at least in part on the second data.
6. The portable communication device of claim 1 , wherein the processor is further configured to:
display an input window as at least part of the second user interface.
7. The portable communication device of claim 1 , wherein the processor is further configured to:
receive, as the second user input, a voice input or a touch input.
8. The portable communication device of claim 1 , wherein the processor is further configured to:
receive, via the microphone, a voice input as the second user input; and
display a text corresponding to at least part of the voice input in the second user interface.
9. The portable communication device of claim 1 , wherein the processor is further configured to:
transmit, using the communication circuitry, third data corresponding to the second user input to the server;
receive, using the communication circuitry, fourth data in response to the third data from the server; and
identify the plurality of messenger applications based at least in part on the fourth data.
10. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving a first user input requesting to transmit at least one image, via a microphone operatively coupled with the one or more processors;
transmitting information corresponding to the first user input to a server;
receiving, from the server, a sequence of a plurality of operations based on the first user input, the sequence including a first subsequence corresponding to a first set of operations and a second subsequence corresponding to a second set of operations, the first set of operations including a first operation which is performable via an image browsing application, and the second set of operations including a second operation which is performable via a plurality of messenger applications;
performing the first set of operations in the first subsequence by executing the image browsing application, the performing of the first set of operations including displaying, via a touchscreen display operatively coupled with the one or more processors, a first user interface indicating the at least one image to be transmitted;
at least temporarily while at least part of the first set of operations is performed, displaying, via the touchscreen display, a second user interface requesting a second user input to select a messenger application among the plurality of messenger applications;
receiving the second user input selecting the messenger application among the plurality of messenger applications via the touchscreen display or the microphone;
determining, based on the second user input, one of the plurality of messenger applications to perform the second set of operations; and
displaying a third user interface and transmitting the at least one image using the selected messenger application by performing the second set of operations in the second subsequence via the determined one of the plurality of messenger applications, the performing of the second set of operations including entering the at least one image in an input interface of the third user interface.