Granted Patent B2

US 12,308,016 · App. 17/666,900 · Granted May 20, 2025

Inventors: Hoseon Shin (Gyeonggi-do, KR); Chulmin Lee (Gyeonggi-do, KR); Taegu Kim (Gyeonggi-do, KR); Kyounggu Woo (Gyeonggi-do, KR); Youngwoo Lee (Gyeonggi-do, KR)

Assignee: Samsung Electronics Co., Ltd

G10L15/08G10L2015/088

Abstract

Claims (78)

1. An electronic device comprising:

a communication circuit;

a plurality of microphones;

at least one sensor configured to detect movement of the electronic device;

a speaker;

at least one processor; and

memory storing instructions,

wherein the instructions that, when executed by the at least one processor individually or collectively, cause the electronic device to:

output audio through the speaker based on data received from an external electronic device through the communication circuit;

identify an utterance including a specified keyword received through at least one of the plurality of microphones;

identify whether a user of the electronic device has turned their head based on a pattern of values received by the at least one sensor; and

based on identifying the utterance including the specified keyword and the user turning their head, decrease a volume of the output audio through the speaker and perform an operation for providing a speech of the user of the electronic device and a speech of a conversant with the user of the electronic device based on at least some of ambient sounds received through at least some of the plurality of microphones, from the user's preset mouth direction, forward direction, and a direction corresponding to an angle at which the head is turned,

wherein providing the speech of the user and the speech of the conversant with the user further comprises generating a new speech model by learning with a speaker embedding for the user, a speaker embedding for the conversant with the user, and at least one other speaker embedding for at least one other speaker contemporaneously speaking, the new speech model providing unique identifications for the conversant with the user, and each of the at least one other speaker.

2. The electronic device of claim 1 ,

wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:

store a plurality of keywords from the external electronic device or sounds corresponding to the plurality of keywords in the memory;

receive the utterance through the at least one of the plurality of microphones during outputting of the audio; and

identify whether the utterance includes the specified keyword based on the received utterance and at least one of the plurality of keywords or the sounds corresponding to the plurality of keywords.

3. The electronic device of claim 2 , wherein the plurality of keywords includes at least one first keyword and at least one second keyword, and

wherein the at least one first keyword is a name of the user, and the at least one second keyword is generated based on the at least one first keyword.

4. The electronic device of claim 1 , wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:

identify whether the utterance has been spoken by the user, based on identifying the utterance including the specified keyword, and

based on identifying that the utterance has been spoken by the user, decrease the volume of the audio output through the speaker and perform the operation for providing the speech of the user of the electronic device and the speech of the conversant with the user.

5. The electronic device of claim 4 , further comprising a sensor,

wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:

identify at least one specified value by using the sensor based on identifying the utterance including the specified keyword, the at least one specified value indicating that the utterance has been spoken by the user; and

identify that the utterance has been spoken by the user, based on identifying the at least one specified value.

6. The electronic device of claim 4 , wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:

generate first feature information about at least one speech of the user, before identifying the utterance including the specified keyword;

compare feature information about the utterance received through at least one of the plurality of microphones with the first feature information, based on identifying the utterance including the specified keyword; and

identify that the utterance has been spoken by the user based on a result of the comparison.

7. The electronic device of claim 1 , wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:

receive the speech of the conversant with the user through at least one of the plurality of microphones, based on identifying the utterance including the specified keyword;

when the speech of the conversant with the user has been received for a specified time period, generate a speaker embedding for the conversant with the user based on the speech of the conversant with the user received for the specified time period; and

obtain the speech of the user and the speech of the conversant with the user based on an obtained at least one speech model, and output the speech of the user and the speech of the conversant with the user.

8. The electronic device of claim 7 ,

wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:

compare at least one speaker embedding for at least one person pre-stored in the memory with the speaker embedding for the conversant with the user; and

when the conversant with the user is identified as different from the at least one person based on a result of the comparison, obtain the at least one speech model.

9. The electronic device of claim 8 , wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to generate the at least one speech model by performing training by using the speaker embedding for the user and the speaker embedding for the conversant with the user as input data and using a first identifier corresponding to the user and a second identifier corresponding to the conversant with the user as output data, and

wherein the at least one speech model is configured to output the first identifier or the second identifier in response to input of the speaker embedding for the user or the speaker embedding for the conversant with the user.

10. The electronic device of claim 9 , wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:

receive the ambient sounds through at least one of the plurality of microphones after the at least one speech model is generated;

separate a plurality of speeches from the received ambient sounds, the plurality of speeches corresponding to a plurality of persons;

generate speaker embeddings for the plurality of persons based on the plurality of speeches;

obtain a plurality of identifiers corresponding to the plurality of speeches based on the generated speaker embeddings being input to the at least one speech model; and

obtain a speech of the user related to the first identifier and a speech of the conversant with the user related to the second identifier among the plurality of speeches.

11. The electronic device of claim 10 , wherein the at least one processor is configured to post-process the obtained speeches of the user and the conversant with the user and output the post-processed speeches of the user and the conversant with the user through the speaker.

12. The electronic device of claim 10 , wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:

further identify a third identifier of a person selected by the user;

further obtain a third speech corresponding to the third identifier among the plurality of speeches; and

output the speech of the user, the speech of the conversant with the user, and the third speech through the speaker.

13. The electronic device of claim 1 , wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:

based on identifying the utterance including the specified keyword, stop output of anti-noise generated by an active noise cancellation circuit of the electronic device through the speaker.

14. A method of operating an electronic device, the method comprising:

outputting audio through a speaker based on data received from an external electronic device through a communication circuit of the electronic device;

identifying an utterance including a specified keyword received through at least some of a plurality of microphones of the electronic device;

identifying whether a user of the electronic device has turned their head based on a pattern of values received by at least one sensor; and

based on identifying the utterance including the specified keyword and the user turning their head, decreasing volume of the audio output through the speaker and performing an operation for providing a speech of the user of the electronic device and a speech of a conversant with the user of the electronic device based on at least some of ambient sounds received through at least one of the plurality of microphones, from the user's preset mouth direction, forward direction, and a direction corresponding to an angle at which the head is turned,

wherein providing the speech of the user and the speech of the conversant with the user further comprises generating a new speech model by learning with a speaker embedding for the user, a speaker embedding for the conversant with the user, and at least one other speaker embedding for at least one other speaker contemporaneously speaking, the new speech model providing unique identifications for the conversant with the user, and each of the at least one other speaker.

15. The method of claim 14 , further comprising:

storing a plurality of keywords from the external electronic device or sounds corresponding to the plurality of keywords in a memory of the electronic device;

receiving the utterance through the at least one of the plurality of microphones during outputting of the audio; and

identifying whether the utterance includes the specified keyword based on the received utterance and at least one of the plurality of keywords or the sounds corresponding to the plurality of keywords.

16. The method of claim 15 , wherein the plurality of keywords include at least one first keyword and at least one second keyword, and

wherein the at least one first keyword is a name of the user, and the at least one second keyword is generated based on the at least one first keyword.

17. The method of claim 14 , further comprising:

identifying whether the utterance has been spoken by the user, based on identifying the utterance including the specified keyword; and

based on identifying that the utterance has been spoken by the user, decreasing the volume of the audio output through the speaker and perform the operation for providing the speech of the user of the electronic device and the speech of the conversant with the user.

18. The method of claim 17 , further comprising:

identifying at least one specified value by using a sensor of the electronic device based on identifying the utterance including the specified keyword, the at least one specified value indicating that the utterance has been spoken by the user; and

identifying that the utterance has been spoken by the user, based on identifying the at least one specified value.

19. A non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by at least one processor of an electronic device, cause the electronic device to perform operations, the operations comprising:

outputting audio through a speaker based on data received from an external electronic device through a communication circuit of the electronic device;

identifying an utterance including a specified keyword received through at least some of a plurality of microphones of the electronic device;

identifying whether a user of the electronic device has turned their head based on a pattern of values received by at least one sensor; and

based on identifying the utterance including the specified keyword and the user turning their head, decreasing volume of the audio output through the speaker and performing an operation for providing a speech of the user of the electronic device and a speech of a conversant with the user of the electronic device based on at least some of ambient sounds received through at least one of the plurality of microphones, from the user's preset mouth direction, forward direction, and a direction corresponding to an angle at which the head is turned,

wherein providing the speech of the user and the speech of the conversant with the user further comprises generating a new speech model by learning with a speaker embedding for the user, a speaker embedding for the conversant with the user, and at least one other speaker embedding for at least one other speaker contemporaneously speaking, the new speech model providing unique identifications for the conversant with the user, and each of the at least one other speaker.

Assignments (1)

Priority Claims (1)

Continuity (2)

References Cited (30)