Method and system for multimodal interaction with sound device connected to network

1. A multimodal interaction method of a multimodal interaction system including an electronic device, comprising:

outputting, by a processor of the electronic device, audio information for playing content through a voice-based interface included in the electronic device;

receiving, by the processor, a voice input of an utterer responsive to the output audio information through the voice-based interface;

generating, by the processor, location information associated with the voice input of the utterer;

determining, an operation associated with the playing of the content based on the voice input responsive to the output audio information and the location information associated with the voice input; and

performing, by the electronic device, the determined operation associated with the playing of the content.

2. The multimodal interaction method of claim 1 , wherein the location information associated with the voice input comprises at least one of a relative location or orientation of the utterer relative to the electronic device that is measured at a point in time or during a period of time associated with the reception of the voice input, whether the relative location or orientation is changed, a level of change in the relative location or orientation, and an orientation in which the relative location or orientation is changed.

3. The multimodal interaction method of claim 1 , wherein the generating of the location information comprises generating the location information associated with the voice input based on a phase shift of the voice input that is input to a plurality of microphones included in the voice-based interface.

4. The multimodal interaction method of claim 1 , wherein the electronic device comprises at least one of a camera and a sensor, and

the generating of the location information comprises generating the location information associated with the voice input based on an output value of at least one of the camera and the sensor in response to receiving the voice input.

5. The multimodal interaction method of claim 1 , wherein the determining of the operation associated with the playing of the content comprises integrating at least one of a tone of sound corresponding to the voice input, a pitch of the sound, and a command extracted by analyzing the voice input and the location information associated with the voice input.

6. The multimodal interaction method of claim 1 , further comprising:

receiving a measurement value that is measured by a sensor of a peripheral device interacting with the electronic device in association with the voice input, from the peripheral device,

wherein the determining of the operation associated with the playing of the content comprises using the received measurement value.

7. The multimodal interaction method of claim 1 , further comprising:

receiving a measurement value that is measured by a sensor of a peripheral device interacting with the electronic device regardless of the voice input, from the peripheral device; and

changing a setting associated with the playing of the content based on the received measurement value.

8. The multimodal interaction method of claim 1 , wherein the audio information comprises information that requires a change in a location of the utterer, and

the determining of the operation associated with the playing of the content depends on whether the voice input and the location information associated with the voice input meet a condition corresponding to the required information.

9. The multimodal interaction method of claim 1 , wherein the content is provided through an external server communicating with the electronic device over a network, and

the determining of the operation associated with the playing of the content comprises:

transmitting the voice input and the location information associated with the voice input to the external server over the network;

receiving, from the external server, operation information that is generated by the external server based on the voice input and the location information associated with the voice input; and

determining the operation associated with the playing of the content based on the received operation information.

10. A non-transitory computer-readable storage medium storing a program, which when executed by a processor, causing the processor to perform the multimodal interaction method of claim 1 .

11. A multimodal interaction system comprising:

a voice-based interface; and

at least one processor configured to execute computer-readable instructions,

wherein the at least one processor is configured to

output audio information for playing content through the voice-based interface,

receive a voice input of an utterer responsive to the output audio information through the voice-based interface,

generate location information associated with the voice input of the utterer,

determine an operation associated with the playing of the content based on the voice input responsive to the output audio information and the location information associated with the voice input, and

performing the determined operation associated with the playing of the content.

12. The multimodal interaction system of claim 11 , wherein the at least one processor is configured to generate the location information associated with the voice input based on a phase shift of the voice input that is input to a plurality of microphones included in the voice-based interface.

13. The multimodal interaction system of claim 11 , further comprising at least one of a camera and a sensor,

wherein the at least one processor is configured to generate the location information associated with the voice input based on an output value of at least one of the camera and the sensor in response to receiving the voice input.

14. The multimodal interaction system of claim 11 , wherein the at least one processor is configured to determine the operation associated with the playing of the content by integrating at least one of a tone of sound corresponding to the voice input, a pitch of the sound, and a command extracted by analyzing the voice input and the location information associated with the voice input.

15. The multimodal interaction system of claim 11 , wherein the at least one processor is configured to receive a measurement value that is measured by a sensor of a peripheral device interacting with the multimodal interaction system in association with the voice input, from the peripheral device, and

determine the operation associated with the playing of the content by using the received measurement value.

16. The multimodal interaction system of claim 11 , wherein the at least one processor is configured to

receive a measurement value that is measured by a sensor of a peripheral device interacting with the multimodal interaction system regardless of the voice input, from the peripheral device, and

change a setting associated with the playing of the content based on the measurement value.

17. The multimodal interaction system of claim 11 , wherein the audio information comprises information that requires an utterance of the utterer and a change in a location of the utterer, and

the at least one processor is configured to determine the operation associated with the playing of the content depending on whether the voice input and the location information associated with the voice input meet a condition corresponding to the required information.

18. The multimodal interaction system of claim 11 , wherein the content is provided through an external server performing communication over a network, and

to determine the operation associated with the playing of the content, the at least one processor is configured to

transmit the voice input and the location information associated with the voice input to the external server over the network,

receive, from the external server, operation information that is generated by the external server based on the voice input and the location information associated with the voice input, and

determine the operation associated with the playing of the content based on the received operation information.