Electronic apparatus and controlling method thereof

1. An electronic apparatus comprising:

a microphone;

a camera;

a memory configured to store at least one command; and

at least one processor configured to be connected to the microphone, the camera, and the memory and control the electronic apparatus,

wherein the at least one processor is further configured, by executing the at least one command, to:

receive a first voice input from a first user through the microphone, using a first natural language understanding model at the electronic apparatus to identify a first category information of the first voice input and provide a response to the first voice input,

receive a second voice input through the microphone while the response to the first voice input is being provided, analyze an image captured by the camera and determine whether the second voice input is uttered by the first user based on the captured image,

in response to determining that the second voice input is uttered by the first user, stop the providing the response to the first voice input and using the first natural language understanding model to identify second category information in the second voice input,

determine whether the first category in the first voice input is being related to the second category in the second voice input,

in response to the first category information in the first voice input is being related to the second category information in the second voice input, use the first natural language understanding model to perform natural language understanding on the second voice input, and

in response to the first category information in the first voice input is being unrelated to the second category information in the second voice input, control a communication interface to transmit the second voice input to a server and using a second natural language understanding model at the server to perform natural language understanding on the second voice input.

2. The electronic apparatus of claim 1 , wherein the at least one processor is further configured to:

based on a third voice input including a wake-up word being input by the first user, recognize the wake-up word and register the first user included in an image captured by the camera;

extract an area including the registered user in the image captured by the camera while the response to the first voice input is being provided; and

determine whether the second voice input uttered by the registered user is in an audio signal by analyzing the extracted area including the registered user.

3. The electronic apparatus of claim 2 , wherein the at least one processor is further configured to:

extract a lip area of the registered user in the image captured by the camera; and

determine whether the second voice input uttered by the registered user is in the audio signal by analyzing whether there is a movement of the extracted lip area of the registered user.

4. The electronic apparatus of claim 1 , wherein the at least one processor is further configured to, based on an audio signal being input while the response to the first voice input is being provided, store the audio signal in a buffer without regard to whether the second voice input uttered by the first user is in the audio signal.

5. The electronic apparatus of claim 4 , wherein the at least one processor is further configured to:

based on determining that the second voice input uttered by the first user is in the audio signal, obtain and provide the response to the second voice input based on the audio signal stored in the buffer; and

based on determining that the second voice input uttered by the first user is not in the audio signal, ignore the audio signal stored in the buffer.

6. The electronic apparatus of claim 1 , wherein the at least one processor is further configured to:

based on determining that the second voice input uttered by the first user is in an audio signal, identify a sentence type of the second voice input; and

perform natural language understanding regarding the second voice input, based on the identified sentence type.

7. The electronic apparatus of claim 6 , wherein the at least one processor is further configured to, based on the identified sentence type being interrogative, control to transmit information regarding the second voice input to a server providing the second natural language understanding model.

8. The electronic apparatus of claim 1 , wherein the at least one processor is further configured to:

based on the first voice input being input, store the first voice input and information regarding the response to the first voice input in the memory as conversation history information of the first user; and

based on determining that the second voice input uttered by the first user is in an audio signal, obtain and provide the response to the second voice input based on the second voice input and the conversation history information.

9. The electronic apparatus of claim 1 , wherein the at least one processor is further configured to:

determine a priority of the second voice input; and

based on a determination that the priority of the second voice input is higher than a threshold value, control to transmit information regarding the second voice input to a server providing the second natural language understanding model.

10. A controlling method of an electronic apparatus, the method comprising:

receiving a first voice input from a first user through a microphone, using a first natural language understanding model at the electronic apparatus to identify a first category information of the first voice input and provide a response to the first voice input;

receiving a second voice input through the microphone while the response to the first voice input is being provided, analyze an image captured by a camera and determine whether the second voice input is uttered by the first user based on the captured image;

in response to determining that the second voice input is uttered by the first user, stopping the providing the response to the first voice input and using the first natural language understanding model to identify second category information in the second voice input;

determining whether the first category in the first voice input is being related to the second category in the second voice input;

in response to the first category information in the first voice input is being related to the second category information in the second voice input, using the first natural language understanding model to perform natural language understanding on the second voice input; and

in response to the first category information in the first voice input is being unrelated to the second category information in the second voice input, transmitting the second voice input to a server and using a second natural language understanding model at the server to perform natural language understanding on the second voice input.

11. The method of claim 10 , further comprising:

based on a third voice input including a wake-up word being input by the first user, recognizing the wake-up word and registering the first user included in an image captured by the camera,

wherein the determining that the second voice input uttered by the first user is in an audio signal comprises:

extracting an area including the registered user in the image captured by the camera while the response to the first voice input is being provided; and

determining whether the second voice input uttered by the registered user is in the audio signal by analyzing the extracted area including the registered user.

12. The method of claim 11 , wherein the determining that the second voice input uttered by the first user is in the audio signal further comprises:

extracting a lip area of the registered user in the image captured by the camera; and

determining whether the second voice input uttered by the registered user is in the audio signal by analyzing whether there is a movement of the extracted lip area of the registered user.

13. The method of claim 10 , further comprising:

based on an audio signal being input while the response to the first voice input is being provided, storing the audio signal in a buffer without regard to whether the second voice input uttered by the first user is in the audio signal.

14. The method of claim 13 , wherein the providing of the response to the second voice input comprises:

based on determining that the second voice input uttered by the first user is in the audio signal, obtaining a response to the second voice input based on the audio signal stored in the buffer, and

based on determining that the second voice input uttered by the first user is not in the audio signal, ignoring the audio signal stored in the buffer.

15. The method of claim 10 , wherein the providing of the response to the second voice input comprises:

based on determining that the second voice input uttered by the first user is in an audio signal, identifying a sentence type of the second voice input; and

performing natural language understanding regarding the second voice input based on the identified sentence type.

16. The method of claim 10 , further comprising:

based on the first voice input being input, storing the first voice input and information regarding the response to the first voice input in a memory as conversation history information of the first user,

based on determining that the second voice input uttered by the first user is in an audio signal, obtaining and providing the response to the second voice input based on the second voice input and the conversation history information.