IP Library Granted Patent US 11,430,438
Granted Patent B2
US 11,430,438 · App. 16/815,108 · Granted Aug 30, 2022

Electronic device providing response corresponding to user conversation style and emotion and method of operating same

Inventors: Piotr Andruszkiewicz (Plac Europejski, PL); Tomasz Latkowski (Plac Europejski, PL); Kamil Herba (Plac Europejski, PL); Maciej Pienkosz (Plac Europejski, PL); Iryna Orlova (Plac Europejski, PL); Jakub Staniszewski (Plac Europejski, PL); Krystian Koziel (Plac Europejski, PL)
Assignee: SAMSUNG ELECTRONICS CO., LTD.
G10L15/22G10L15/1815G10L15/24G10L15/30G10L25/63G10L2015/223G10L2015/226
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,430,438
App. No.
16/815,108
Granted
Aug 30, 2022
Kind
B2
Abstract

An electronic device includes a microphone, a communication circuit, and a processor configured to obtain a user's utterance through the microphone, transmit first information about the utterance through the communication circuit to an external server for at least partially automatic speech recognition (ASR) or natural language understanding (NLU), obtain a second text from the external server through the communication circuit, the second text being a text resulting from modifying at least part of a first text included in a neutral response to the utterance based on parameters corresponding to the user's conversation style and emotion identified based on the first information, and provide a voice corresponding to the second text or a message including the second text in response to the utterance.

Claims (44)

1. An electronic device, comprising:

a microphone;

a communication circuit; and

a processor configured to:

obtain an utterance of a user through the microphone,

transmit first information about the utterance through the communication circuit to an external server for automatic speech recognition (ASR) and natural language understanding (NLU),

obtain a second text from the external server through the communication circuit, wherein a neutral response for the first information and the second text are generated sequentially, wherein the neutral response is generated based on content of the utterance which is recognized based on the first information by performing the ASR and the NLU, and wherein the second text is a text resulting from modifying at least part of a first text included in the neutral response corresponding to the utterance based on parameters corresponding to a conversation style of the user and an emotion of the user that are identified based on the first information, and

provide a voice corresponding to the second text or a message including the second text in response to the utterance while performing a function corresponding to the utterance.

2. The electronic device of claim 1 , wherein a parameter for the emotion of the user is identified based on at least one of a text, voice, sound volume, image, or video for the utterance.

3. The electronic device of claim 1 , wherein a parameter for the conversation style of the user is identified based on at least one of a content of the utterance, intonation of the utterance or a conversation history related to the utterance.

4. The electronic device of claim 1 , wherein the second text includes a text resulting from further modifying the at least part of the first text based on a parameter which is based on at least one of biometric information of the user, a location of a terminal corresponding to the user, or acceleration information about the terminal.

5. The electronic device of claim 1 , wherein the second text further includes a new text corresponding to the conversation style and the emotion.

6. The electronic device of claim 1 , wherein the processor is configured to:

obtain a third text displayed on a display of the electronic device,

transmit, through the communication circuit, second information about the third text to the external server to recognize the third text,

obtain a fifth text from the external server through the communication circuit, the fifth text being a text resulting from modifying at least part of a fourth text included in a neutral response to the third text based on parameters corresponding to the conversation style of the user and the emotion of the user that are identified based on the second information, and

provide a voice corresponding to the fifth text or a message including the fifth text in response to the third text.

7. The electronic device of claim 6 , wherein a parameter for the emotion of the user is identified based on at least one of at least one text included in the third text, an image related to the third text or video related to the third text.

8. The electronic device of claim 6 , wherein a parameter for the conversation style of the user is identified based on at least one of a content of the third text or a conversation history related to the third text.

9. The electronic device of claim 1 , wherein the conversation style and the emotion are selected from among a plurality of predetermined conversation styles and emotions.

10. A method for operating an electronic device, the method comprising:

obtaining an utterance of a user through a microphone of the electronic device;

transmitting first information about the utterance through a communication circuit of the electronic device to an external server for ASR and NLU;

obtaining a second text from the external server through the communication circuit, wherein a neutral response for the first information and the second text are generated sequentially, wherein the neutral response is generated based on content of the utterance which is recognized based on the first information by performing the ASR and the NLU, and wherein the second text is a text resulting from modifying at least part of a first text included in the neutral response corresponding to the utterance based on parameters corresponding to a conversation style of the user and an emotion of the user that are identified based on the first information; and

providing a voice corresponding to the second text or a message including the second text in response to the utterance while performing a function corresponding to the utterance.

11. The method of claim 10 , wherein a parameter for the emotion of the user is identified based on at least one of a text, voice, sound volume, image, or video for the utterance.

12. The method of claim 10 , wherein a parameter for the conversation style of the user is identified based on at least one of a content of the utterance, intonation of the utterance or a conversation history related to the utterance.

13. The method of claim 10 , wherein the second text includes a text resulting from further modifying the at least part of the first text based on a parameter which is based on at least one of biometric information of the user, a location of a terminal corresponding to the user, or acceleration information about the terminal.

14. The method of claim 10 , wherein the second text further includes a new text corresponding to the conversation style and the emotion.

15. The method of claim 10 , further comprising:

obtaining a third text displayed on a display of the electronic device;

transmitting, through the communication circuit, second information about the third text to the external server to recognize the third text;

obtaining a fifth text from the external server through the communication circuit, the fifth text being a text resulting from modifying at least part of a fourth text included in a neutral response to the third text based on parameters corresponding to the conversation style of the user and the emotion of the user that are identified based on the second information; and

providing a voice corresponding to the fifth text or a message including the fifth text in response to the third text.

16. The method of claim 15 , wherein a parameter for the emotion of the user is identified based on at least one of at least one text included in the third text, an image related to the third text or video related to the third text.

17. The method of claim 15 , wherein a parameter for the conversation style of the user is identified based on at least one of a content of the third text or a conversation history related to the third text.

18. An electronic device, comprising:

a microphone; and

a processor configured to:

obtain an utterance of a user through the microphone,

obtain a neutral first response corresponding to the utterance by performing ASR and NLU,

identify information about a conversation style of the user and an emotion of the user based on the utterance, obtain a second response including a second text resulting from modifying at least part of a first text included in the neutral first response based on the identified information, wherein the neutral first response and the second response are generated sequentially, and wherein the neutral first response is generated based on content of the utterance which is recognized by performing the ASR and the NLU, and

provide the second response through a voice or a message in response to the utterance while performing a function corresponding to the utterance.

19. The electronic device of claim 18 , wherein the second text further includes a new text corresponding to the conversation style and the emotion.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 11, 2020
From: ANDRUSZKIEWICZ, PIOTR; LATKOWSKI, TOMASZ; HERBA, KAMIL; PIENKOSZ, MACIEJ; ORLOVA, IRYNA; STANISZEWSKI, JAKUB; KOZIEL, KRYSTIAN
To: SAMSUNG ELECTRONICS CO., LTD.
Reel/Frame 052144/0823 →
Priority Claims (1)
KR 10-2019-0032836 · Mar 22, 2019 · national
Continuity (1)
Related Publication 20200302927A1 · Sep 24, 2020