IP Library › Granted Patent US 11,170,580
Granted Patent B2
US 11,170,580 · App. 16/474,407 · Granted Nov 9, 2021

Information processing device, information processing method, and recording medium

Inventors: Yo Nonoyama (Kanagawa, JP); Emily Rei Worung (Kanagawa, JP)
Assignee: SONY CORPORATION
G06T19/20G06F3/011G06F3/013G06F3/016G06K9/00355G06K9/00362G06T2219/2016
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,170,580
App. No.
16/474,407
Granted
Nov 9, 2021
Kind
B2
Abstract

Provided is an information processing device including a recognition unit that recognizes a manipulation of a user wearing a display device, and an output control unit that controls, while causing the display device to display a screen including a captured image of an observation target, a size of the observation target in the screen, on the basis of a distance between a recognition target that the user manipulates and a criterial position.

Claims (79)

1. An information processing device, comprising:

a recognition unit configured to recognize a first manipulation of a user; and

an output control unit configured to:

control a display device to display a screen including a captured image of an observation target and at least one annotation corresponding to the observation target, wherein

the at least one annotation indicates allowance of the first manipulation to a specific region of the observation target, and

the display device is wearable by the user;

control a size of the observation target in the screen based on a distance between a recognition target and a criterial position, wherein the recognition target is manipulated by the user;

change content of the at least one annotation in the screen based on a change in the size of the observation target in the screen, wherein the at least one annotation in the screen is different from the observation target in the screen; and

change at least one of the content of the at least one annotation or a shape of the at least one annotation in the screen based on emotion of the observation target, wherein the emotion of the observation target is based on the captured image of the observation target.

2. The information processing device according to claim 1 , wherein the output control unit is further configured to:

enlarge the size of the observation target in the screen based on an increase in the distance between the recognition target and the criterial position; and

reduce the size of the observation target in the screen based on a decrease in the distance between the recognition target and the criterial position.

3. The information processing device according to claim 1 , wherein

the distance between the recognition target and the criterial position is a distance in a criterial direction between the criterial position and the recognition target, and

the output control unit is further configured to control, based on a position of the recognition target in a direction orthogonal to the criterial direction, a central position of the control of the size of the observation target in the screen.

4. The information processing device according to claim 3 , wherein the output control unit is further configured to control the display device to display information indicating the central position.

5. The information processing device according to claim 1 , wherein the criterial position is a position of the display device.

6. The information processing device according to claim 1 , wherein the recognition target is a hand of the user in a specific shape.

7. The information processing device according to claim 6 , wherein

the recognition target is the hand of the user with fingers forming a ring, and

the output control unit is further configured to control the size of the observation target in the screen based on an angle of the ring to a vertical direction.

8. The information processing device according to claim 1 , wherein the output control unit is further configured to:

control imaging magnification of the captured image; and

control the size of the observation target in the screen based on the imaging magnification.

9. The information processing device according to claim 1 , wherein the output control unit is further configured to:

recognize the first manipulation to the specific region of the observation target; and

control an output device to output tactile information based on the recognized first manipulation to the specific region of the observation target, wherein the output tactile information imitates a tactile sensation acquired at a touch of the observation target.

10. The information processing device according to claim 9 , wherein

the output control unit is further configured to control display of an image indicating a hand of the user in a superimposition position on the observation target in the screen, and

the hand of the user is superimposed on the observation target in the screen based on the recognized first manipulation to the specific region of the observation target.

11. The information processing device according to claim 9 , wherein

the observation target is an animal, and

the tactile information includes at least one of heartbeat, body temperature, or a feel of a body surface of the observation target.

12. The information processing device according to claim 9 , wherein the output control unit is further configured to control the display of the at least one annotation indicating the allowance of the first manipulation to the specific region in association with the specific region.

13. The information processing device according to claim 9 , wherein

the output device is wearable on a hand of the user, and

the output device includes at least one of a vibration generation device, a thermal generation device, or a feel generation device.

14. The information processing device according to claim 1 , wherein

the output control unit is further configured to control the display device to display a plurality of display objects,

each display object of the plurality of display objects corresponds to a respective candidate image capturing device of a plurality of candidate image capturing devices,

at least one candidate image capturing device of the plurality of candidate image capturing devices is a source of provision of the captured image, and

the recognition unit is further configured to recognize, as a selection manipulation to the at least one candidate image capturing device, a second manipulation to a display object of the plurality of display objects.

15. The information processing device according to claim 14 , wherein

the output control unit is further configured to control display of each display object of the plurality of display objects at a position corresponding to installed position of the respective candidate image capturing device in the screen.

16. The information processing device according to claim 14 , wherein

the output control unit is further configured to control display of each display object of the plurality of display objects in an arrangement at a height corresponding to an angle of a capturing direction of the respective candidate image capturing device to a vertical direction, and

each display object of the plurality of display objects includes an image captured by the respective candidate image capturing device of the plurality of candidate image capturing devices.

17. The information processing device according to claim 1 , wherein the display device includes:

a first image sensor configured to capture an eye of the user; and

a second image sensor configured to capture the recognition target.

18. An information processing method, comprising:

in an information processing device:

recognizing, by a processor, a manipulation of a user;

controlling, by the processor, a display device to display a screen including a captured image of an observation target and at least one annotation corresponding to the observation target, wherein

the at least one annotation indicates allowance of the manipulation to a specific region of the observation target, and

the display device is wearable by the user;

controlling, by the processor, a size of the observation target in the screen based on a distance between a recognition target and a criterial position, wherein the recognition target is manipulated by the user;

changing, by the processor, content of the at least one annotation in the screen based on a change in the size of the observation target in the screen, wherein the at least one annotation in the screen is different from the observation target in the screen; and

changing, by the processor, at least one of the content of the at least one annotation or a shape of the at least one annotation in the screen based on emotion of the observation target, wherein the emotion of the observation target is based on the captured image of the observation target.

19. A non-transitory computer-readable medium having stored thereon computer-executable instructions which, when executed by a processor, cause the processor to execute operations, the operations comprising:

recognizing a manipulation of a user; and

controlling a display device to display a screen including a captured image of an observation target and at least one annotation corresponding to the observation target, wherein

the at least one annotation indicates allowance of the manipulation to a specific region of the observation target, and

the display device is wearable by the user;

controlling a size of the observation target in the screen based on a distance between a recognition target and a criterial position, wherein the recognition target is manipulated by the user;

changing content of the at least one annotation in the screen based on a change in the size of the observation target in the screen, wherein the at least one annotation in the screen is different from the observation target in the screen; and

changing at least one of the content of the at least one annotation or a shape of the at least one annotation in the screen based on emotion of the observation target, wherein the emotion of the observation target is based on the captured image of the observation target.

20. An information processing device, comprising:

a recognition unit configured to recognize a manipulation of a user; and

an output control unit configured to:

control a display device to display a screen including a captured image of an observation target and at least one annotation corresponding to the observation target, wherein

the at least one annotation indicates allowance of the manipulation to a specific region of the observation target, and

the display device is wearable by the user;

control a size of the observation target in the screen based on

a distance between a hand of the user and a criterial position, and

an angle of a ring, formed by fingers of the hand, to a vertical direction;

control content of the at least one annotation based on the size of the observation target in the screen; and

control at least one of the content of the at least one annotation, a shape of the at least one annotation, or a display position of the at least one annotation based on emotion of the observation target, wherein

the emotion of the observation target is based on the captured image of the observation target.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 27, 2019
From: NONOYAMA, YO; WORUNG, EMILY REI
To: SONY CORPORATION
Reel/Frame 049614/0203 →
Priority Claims (1)
JP JP2017-027238 · Feb 16, 2017 · national
Continuity (1)
Related Publication 20200258314A1 · Aug 13, 2020
Cited By (1)
US 12,462,717