Granted Patent B2

US 12,536,992 · App. 17/575,195 · Granted Jan 27, 2026

Inventors: Hojung Lee (Suwon-si, KR); Hyungtak Choi (Suwon-si, KR); Munjo Kim (Suwon-si, KR)

Assignee: SAMSUNG ELECTRONICS CO., LTD.

G10L15/08G06F3/04842G06F3/167G10L2015/088

Abstract

Claims (72)

1 . A method, performed by an electronic device, of providing a voice recognition service, the method comprising:

receiving a voice command of a user while one or more objects are displayed on a screen of the electronic device;

based on the receiving the voice command, identifying the one or more objects displayed on the screen;

obtaining text from the voice command;

determining, from the text obtained from the voice command, a type of an utterance intent of the user as actionable or accessible, or not actionable or accessible;

identifying types of the one or more objects as text or interpretable, and as selectable or non-selectable by a user input to the electronic device;

based on the type of the utterance intent being actionable or accessible, interpreting the text obtained from the voice command in a first mode by performing natural language understanding with respect to the text based on a data structure generated based on the types of the one or more objects displayed on the screen, and by:

based on the type of the utterance intent being actionable, preferentially referring to the one or more objects that are selectable; and

based on the type of the utterance intent being accessible, preferentially referring to the one or more objects that are text;

selecting a selectable object from the one or more objects displayed on the screen, based on a result of the interpreting of the text;

performing an operation related to a selected object;

receiving a second voice command of the user while the one or more objects are displayed on the screen of the electronic device;

based on the receiving the second voice command, identifying the one or more objects displayed on the screen;

obtaining text from the second voice command;

determining, from the text obtained from the second voice command, the type of the utterance intent of the user as actionable or accessible, or not actionable or accessible;

identifying types of the one or more objects as text or interpretable, and as selectable or non-selectable by a user input to the electronic device;

based on the type of the utterance intent being not actionable or not accessible, interpreting the text obtained from the second voice command in a second mode by performing the natural language understanding on the text without using the data structure generated based on the types of the one or more objects; and

generating a response message in a form of at least one of voice, text, or video, to the result of interpreting the text from the second voice command.

2 . The method of claim 1 , further comprising:

generating the data structure by:

identifying the types of the one or more objects through character recognition based on image processing with respect to the one or more objects or through meta data reading with respect to an application that provides the one or more objects;

determining priorities of the one or more objects, based on the types of the one or more objects; and

generating the data structure in a tree form indicating a relationship between the one or more objects and terms related to the one or more objects.

3 . The method of claim 2 , wherein the terms related to the one or more objects are obtained from at least one of text information included in the one or more objects or attribute information of the one or more objects included in the meta data of the application that provides the one or more objects.

4 . The method of claim 2 , wherein the one or more objects comprise at least one of image information or text information,

the image information is displayed on an object layer of the screen,

the text information is displayed on a text layer of the screen, and

the types of the one or more objects are identified based on the object layer and the text layer.

5 . The method of claim 2 , wherein the identifying the types of the one or more objects further comprises identifying whether the one or more objects include text information.

6 . The method of claim 1 , further comprising:

generating the data structure,

wherein the identifying the one or more objects displayed on the screen comprises:

determining priorities of the one or more objects of the data structure, based on the type of the utterance intent and the types of the one or more objects.

7 . The method of claim 1 , wherein the determining whether to operate in the first mode or in the second mode is also based on whether an activation word is included in the text.

8 . The method of claim 1 , wherein the result of the interpreting the text comprises information about at least one of the utterance intent of the user, the selected object, or a function to be executed by the electronic device in relation to the selected object.

9 . The method of claim 1 , wherein the operation related to the selected object comprises at least one of playing a video related to the selected object, enlarging and displaying an image or text related to the selected object, or outputting an audio based on the text included in the selected object.

10 . The method of claim 1 , wherein the performing the operation related to the selected object comprises generating and outputting the response message related to the selected object, based on the voice command of the user.

11 . An electronic device for providing a voice recognition service, the electronic device comprising:

a display;

a microphone;

a memory storing one or more instructions; and

at least one processor configured to execute one or more instructions stored in the memory to:

receive a voice command of a user via the microphone while one or more objects are displayed on a screen of the display;

based on receipt of the voice command, identify the one or more objects displayed on the screen;

obtain text from the voice command;

determine, from the text obtained from the voice command, a type of an utterance intent of the user as (i) actionable or accessible or (ii) not actionable or not accessible;

identify types of the one or more objects as text or interpretable, and as selectable or non-selectable by a user input to the electronic device;

based on the type of the utterance intent being actionable or accessible, interpret the text obtained from the voice command in a first mode by performing natural language understanding with respect to the text based on a data structure generated based on the types of the one or more objects displayed on the screen, and by:

based on the type of the utterance intent being actionable, preferentially referring to the one or more objects that are selectable; and

based on the type of the utterance intent being accessible, preferentially referring to the one or more objects that are text;

based on the type of the utterance intent being not actionable or not accessible, interpret the text obtained from the voice command in a second mode by performing the natural language understanding on the text without using the data structure generated based on the types of the one or more objects;

select a selectable object from the one or more objects displayed on the screen, based on a result of the interpreting of the text; and

perform an operation related to a selected object.

12 . The electronic device of claim 11 , wherein the at least one processor is further configured to execute the one or more instructions to:

generate the data structure;

identify the types of the one or more objects through character recognition based on image processing with respect to the one or more objects or through meta data reading with respect to an application that provides the one or more objects; and

determine priorities of the one or more objects, based on the types of the one or more objects, to generate the data structure in a tree form indicating a relationship between the one or more objects and terms related to the one or more objects.

13 . A server for providing a voice recognition service through an electronic device comprising a display, the server comprising:

a communication interface configured to communicate with the electronic device;

a memory storing one or more instructions; and

at least one processor configured to execute one or more instructions stored in the memory to:

receive information about a voice command of a user received through the electronic device while one or more objects are displayed on a screen of the electronic device;

identify the one or more objects displayed on the screen, based on receipt of the voice command;

obtain text from the voice command;

determine, from the text obtained from the voice command, a type of an utterance intent of the user as actionable or accessible;

identify types of the one or more objects as text or interpretable, and as selectable or non-selectable by a user input to the electronic device;

based on the type of the utterance intent being actionable or accessible, interpret the text obtained from the voice command in a first mode by performing natural language understanding with respect to the text based on a data structure generated based on the types of the one or more objects displayed on the screen, and by:

based on the type of the utterance intent being actionable, preferentially referring to the one or more objects that are selectable; and

based on the type of the utterance intent being accessible, preferentially referring to the one or more objects that are text;

based on the type of the utterance intent being not actionable or not accessible, interpret the text obtained from the voice command in a second mode by performing the natural language understanding on the text without using the data structure generated based on the types of the one or more objects;

select a selectable object from the one or more objects displayed on the screen, based on a result of the interpreting of the text; and

control the electronic device to perform an operation related to a selected object.

Assignments (1)

Priority Claims (1)

Continuity (2)

References Cited (35)