Device for outputting object-based feedback, operating method thereof, and recording medium

1 . An electronic device comprising:

memory configured to store computer-executable instructions; and

at least one processor-, comprising processing circuitry, individually and/or collectively, configured to:

obtain, an input of a user;

generate, based on the input, first feedback on the input;

detect an object related the input, in at least one of a display region, a surrounding physical space, or a virtual space;

generate, based on the input and the object, second feedback following the first feedback;

output the first feedback and the second feedback;

classify candidate operations supportable by the object into operation related to the input and operation not related to the input;

while a context corresponding to the input is continued, output a first interface for the operation related to the input, wherein operation not related to the input is excluded from the first interface; and

after the context corresponding to the input is terminated, output a second interface for all candidate operations supportable by the object, including the operation related to the input and the operation not related to the input.

2 . The electronic device of claim 1 , wherein at least one processor, individually and/or collectively, is configured to:

detect a candidate object in at least one of the display region, the surrounding physical space, or the virtual space;

calculate a relevance score of the input and each candidate object;

determine the detected object based on the calculated relevance score;

based on failure of detection of an object associated with the input for a threshold period of time from a time point at which the input is obtained, stop detecting the object;

determine an operation of the object among the candidate operations supportable by the object, based on at least one of the input, the first feedback, or the second feedback; and

output a preview of the object performing the determined operation.

3 . The electronic device of claim 1 , wherein at least one processor, individually and/or collectively, is configured to:

generate the first feedback independently of the object, such that the identical first feedback is generated both when a first object is detected in the display region and when a second object different from the first object is detected in the display region; and

generate the second feedback dependently of the object, such that different second feedbacks are generated when a first object is detected in a display region and when a second object is detected in the display region.

4 . The electronic device of claim 1 , wherein the second feedback

comprises at least one of feedback of suggesting a control command for the object to the user, feedback of displaying an interface for obtaining the control command for the object, feedback of suggesting an action of the user for the object, feedback of providing a preview of the control command for the object, feedback of providing a preview of the action for the object, or feedback of providing additional information for the object.

5 . The electronic device of claim 1 , wherein at least one processor, individually and/or collectively, is configured to:

determine a space where the user is located at a time point at which the input is obtained;

detect the object in the display region while the user is located in the determined space;

while the user is located in the space, determine that the context corresponding to the input is continued and output the first interface; and

based on the user leaving the determined space, determine that the context corresponding to the input is terminated and output the second interface.

6 . The electronic device of claim 1 , wherein at least one processor, individually and/or collectively, is configured to:

detect a motion of the user during a period of time including a time point at which the input is obtained;

while detecting the motion, determine that the context corresponding to the input is continued and output the first interface; and

based on detecting an end of the motion, determine that the context corresponding to the input is terminated and output the second interface.

7 . The electronic device of claim 1 , wherein at least one processor, individually and/or collectively, is configured to:

while the object is detected in the display region, determine that the context corresponding to the input is continued and output the first interface; and

based on failing to detect the object in the display region, determine that the context corresponding to the input is terminated and output the second interface.

8 . The electronic device of claim 1 , wherein at least one processor, individually and/or collectively, is configured to:

obtain an additional input from the user;

based on obtaining the additional input while the context corresponding to the input is continued, analyze the additional input using at least one of the input, the first feedback, or the second feedback along with the additional input; and

based on obtaining the additional input after the context corresponding to the input is terminated, exclude the input, the first feedback, and the second feedback from analyzing the additional input.

9 . The electronic device of claim 1 , wherein at least one processor, individually and/or collectively, is configured to:

obtain a subsequent input of the user, after outputting the first feedback and the second feedback;

determine whether subsequent input is associated with the object;

based on the subsequent input being associated with the object, determine that the context corresponding to the input is continued and output the first interface; and

based on the subsequent input being not associated with the object, determine that the context corresponding to the input is terminated and output the second interface.

10 . The electronic device of claim 1 , wherein at least one processor, individually and/or collectively, is configured to:

detect a plurality of objects related the input;

generate a plurality of second feedbacks respectively corresponding to the plurality of objects;

output a plurality of graphic representations for each of the plurality of objects;

obtain a subsequent input of the user for selecting a graphic representation; and

output a selected second feedback corresponding to the selected graphic representation, among from the plurality of second feedbacks.

11 . A method performed by an electronic device, the method comprising:

obtaining, an input of a user;

generating, based on the input, first feedback on the input;

detecting an object related the input, in at least one of a display region, a surrounding physical space, or a virtual space;

generating, based on the input and the object, second feedback following the first feedback;

outputting the first feedback and the second feedback;

classifying candidate operations supportable by the object into operation related to the input and operation not related to the input;

while a context corresponding to the input is continued, outputting a first interface for the operation related to the input, wherein operation not related to the input is excluded from the first interface; and

after the context corresponding to the input is terminated, outputting a second interface for all candidate operations supportable by the object, including the operation related to the input and the operation not related to the input.

12 . The method of claim 11 ,

wherein the generating of the second feedback comprises:

detecting a candidate object in at least one of the display region, the surrounding physical space, or the virtual space;

calculating a relevance score of the input and each candidate object;

determining the detected object based on the calculated relevance score; and

based on failure of detection of an object associated with the input for a threshold period of time from a time point at which the input is obtained, stopping detecting the object,

wherein the method further comprises:

determining an operation of the object among the candidate operations supportable by the object, based on at least one of the input, the first feedback, or the second feedback; and

outputting a preview of the object performing the determined operation.

13 . The method of claim 11 ,

wherein the generating of the first feedback comprises:

generating the first feedback independently of the object, such that the identical first feedback is generated both when a first object is detected in the display region and when a second object different from the first object is detected in the display region; and

wherein the generating of the second feedback comprises:

generating the second feedback dependently of the object, such that different second feedbacks are generated when a first object is detected in a display region and when a second object is detected in the display region.

14 . The method of claim 11 , wherein the second feedback comprises at least one of feedback of suggesting a control command for the object to the user, feedback of displaying an interface for obtaining the control command for the object, feedback of suggesting an action of the user for the object, feedback of providing a preview of the control command for the object, feedback of providing a preview of the action for the object, or feedback of providing additional information for the object.

15 . The method of claim 11 , wherein the method further comprises:

determining a space where the user is located at a time point at which the input is obtained;

detecting the object in the display region while the user is located in the determined space;

while the user is located in the space, determining that the context corresponding to the input is continued and outputting the first interface; and

based on the user leaving the determined space, determining that the context corresponding to the input is terminated and outputting the second interface.

16 . The method of claim 11 , further comprises:

detecting a motion of the user during a period of time including a time point at which the input is obtained;

while detecting the motion, determining that the context corresponding to the input is continued and outputting the first interface; and

based on detecting an end of the motion, determining that the context corresponding to the input is terminated and outputting the second interface.

17 . The method of claim 11 , further comprises:

while the object is detected in the display region, determining that the context corresponding to the input is continued and outputting the first interface; and

based on failing to detect the object in the display region, determining that the context corresponding to the input is terminated and outputting the second interface.

18 . The method of claim 11 , further comprising:

obtaining an additional input from the user;

based on obtaining the additional input while the context corresponding to the input is continued, analyzing the additional input using at least one of the input, the first feedback, or the second feedback along with the additional input; and

based on obtaining the additional input after the context corresponding to the input is terminated, excluding the input, the first feedback, and the second feedback from analyzing the additional input.

19 . The method of claim 11 , further comprising:

obtaining a subsequent input of the user, after outputting the first feedback and the second feedback;

determining whether subsequent input is associated with the object;

based on the subsequent input being associated with the object, determining that the context corresponding to the input is continued and outputting the first interface; and

based on the subsequent input being not associated with the object, determining that the context corresponding to the input is terminated and outputting the second interface.

20 . A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 11 .