Granted Patent B2

US 10,466,794 · App. 14/775,776 · Granted Nov 5, 2019

Inventors: Yoshinori Maeda (Kanagawa, JP); Jun Yokono (Tokyo, JP); Katsuki Minamino (Tokyo, JP)

Assignee: SONY CORPORATION

G06F3/017G02B27/017G06F3/011G06F3/0482G06F3/04842G02B2027/014G02B2027/0138G02B2027/0178G02B2027/0187

Abstract

Claims (68)

1. An information processing apparatus, comprising:

circuitry configured to:

control a plurality of controllable objects via a user interface, wherein each of the plurality of controllable objects is one of a real object or a virtual object,

associate the user interface with a gesture recognition area that is defined by a detection area of a sensor,

divide the gesture recognition area into a plurality of gesture recognition sub-areas, based on a detection of the plurality of controllable objects controlled by the user interface and a presence of a reference position corresponding to each of the plurality of controllable objects,

assign a priority to each of the plurality of controllable objects based on each of the plurality of controllable objects is one of the real object or the virtual object, wherein:

a first gesture recognition sub-area of the plurality of gesture recognition sub-areas is associated with a first controllable object of the plurality of controllable objects and a second gesture recognition sub-area of the plurality of gesture recognition sub-areas is associated with a second controllable object of the plurality of controllable objects, and

a first priority associated with the first controllable object is greater than a second priority associated with the second controllable object,

adjust a first size of the first gesture recognition sub-area and a second size of the second gesture recognition sub-area based on the first priority and the second priority, respectively, such that the first size is larger than the second size,

detect a gesture within the detection area,

determine one of the plurality of gesture recognition sub-areas in which the gesture is detected, and

execute an operation on one of the plurality of controllable objects based on the detected gesture in the determined one of the plurality of gesture recognition sub-areas.

2. The information processing apparatus according to claim 1 , wherein the circuitry is further configured to:

assign the one of the plurality of controllable objects to the one of the plurality of gesture recognition sub-areas, wherein the one of the plurality of controllable objects is a remote controllable device, and

execute the operation based on a command transmitted by the information processing apparatus to the remote controllable device.

3. The information processing apparatus according to claim 1 , wherein the circuitry is further configured to allocate, for each of the plurality of gesture recognition sub-areas, a sub-region of the plurality of gesture recognition sub-areas based on a center position of each of the plurality of controllable objects corresponding to each of the plurality of gesture recognition sub-areas.

4. The information processing apparatus according to claim 1 , wherein the circuitry is further configured to:

indicate at least one candidate gesture, and

associate a candidate operation for one of the plurality of controllable objects based on the candidate gesture.

5. The information processing apparatus according to claim 1 , wherein the circuitry is further configured to:

display a menu on a display screen of the information processing apparatus, wherein:

the menu includes a plurality of selectable items, and

the menu is associated with the real object of the plurality of controllable objects present in the one of the plurality of gesture recognition sub-areas, and

select one of the plurality of items based on the detected gesture in the one of the plurality of gesture recognition sub-areas assigned to the real object.

6. The information processing apparatus according to claim 1 , wherein the circuitry is further configured to control a display screen to display a visual indication of the one of the plurality of gesture recognition sub-areas based on the detection of the gesture.

7. The information processing apparatus according to claim 6 , wherein the circuitry is further configured to:

control the display screen to display the visual indication of the one of the plurality of gesture recognition sub-areas, and

control the display screen to display a mark on the one of the plurality of controllable objects, wherein the mark on the one of the plurality of controllable objects indicates that the one of the plurality of controllable objects is a target of the operation on the display screen.

8. The information processing apparatus according to claim 1 , further comprising:

a display screen implemented in a head-mounted display unit, wherein the display screen is configured to present the user interface.

9. The information processing apparatus according to claim 1 , wherein:

the plurality of controllable objects includes the virtual object assigned to a third gesture recognition sub-area of the plurality of gesture recognition sub-areas, and the real object assigned to a fourth gesture recognition sub-area of the plurality of gesture recognition sub-areas, and

the circuitry is further configured to control the virtual object and the real object via the user interface, wherein the real object is recognized based on an image captured by the sensor.

10. The information processing apparatus according to claim 1 , wherein the sensor is a camera.

11. The information processing apparatus according to claim 1 , wherein:

the plurality of controllable objects includes the real object, and

the circuitry is further configured to set one of the plurality of gesture recognition sub-areas based on a recognition of a location of the real object.

12. The information processing apparatus according to claim 11 , wherein the circuitry is further configured to superimpose the user interface over a user view of the real object.

13. The information processing apparatus according to claim 1 , wherein the circuitry is further configured to set a boundary of each of the plurality of gesture recognition sub-areas based on relative locations of the plurality of controllable objects associated with each of the plurality of gesture recognition sub-areas.

14. The information processing apparatus according to claim 1 ,

wherein the circuitry is further configured to set the one of the plurality of gesture recognition sub-areas with a priority based on the real object of the plurality of controllable objects is in one of the plurality of gesture recognition sub-areas.

15. The information processing apparatus according to claim 1 , wherein the circuitry is further configured to:

detect a plurality of gestures within the detection area;

merge each of the plurality of gesture recognition sub-areas based on the detection of a first gesture of the plurality of gestures; and

remove at least one of the plurality of gesture recognition sub-areas based on the detection of a second gesture of the plurality of gestures.

16. The information processing apparatus according to claim 1 , wherein the circuitry is further configured to determine the reference position of each of the plurality of controllable objects based on a position of each of the plurality of controllable objects within a display screen.

17. A method, comprising:

controlling, by circuitry of an information processing apparatus, a plurality of controllable objects via a user interface, wherein each of the plurality of controllable objects is one of a real object or a virtual object,

associating the user interface with a gesture recognition area that is defined by a detection area of a sensor,

dividing the gesture recognition area into a plurality of gesture recognition sub-areas, based on a detection of the plurality of controllable objects controlled by the user interface and a presence of a reference position corresponding to each of the plurality of controllable objects;

assigning a priority to each of the plurality of controllable objects based on each of the plurality of controllable objects is one of the real object or the virtual object, wherein:

a first gesture recognition sub-area of the plurality of gesture recognition sub-areas is associated with a first controllable object of the plurality of controllable objects and a second gesture recognition sub-area of the plurality of gesture recognition sub-areas is associated with a second controllable object of the plurality of controllable objects, and

a first priority associated with the first controllable object is greater than a second priority associated with the second controllable object,

adjusting a first size of the first gesture recognition sub-area and a second size of the second gesture recognition sub-area based on the first priority and the second priority, respectively, such that the first size is larger than the second size,

detecting a gesture within the detection area;

determining one of the plurality of gesture recognition sub-areas in which the gesture is detected; and

executing, by the circuitry of the information processing apparatus, an operation on one of the plurality of controllable objects based on the detected gesture in the determined one of the plurality of gesture recognition sub-areas.

18. A non-transitory computer-readable medium having stored thereon computer-executable instructions which when executed by a computer, cause the computer to execute operations, the operations comprising:

controlling a plurality of controllable objects via a user interface, wherein each of the plurality of controllable objects is one of a real object or a virtual object,

associating the user interface with a gesture recognition area that is defined by a detection area of a sensor,

dividing the gesture recognition area into a plurality of gesture recognition sub-areas, based on a detection of the plurality of the controllable objects controlled by the user interface and a presence of a reference position corresponding to each of the plurality of controllable objects;

assigning a priority to each of the plurality of controllable objects based on each of the plurality of controllable objects is one of the real object or the virtual object, wherein:

a first gesture recognition sub-area of the plurality of gesture recognition sub-areas is associated with a first controllable object of the plurality of controllable objects and a second gesture recognition sub-area of the plurality of gesture recognition sub-areas is associated with a second controllable object of the plurality of controllable objects, and

a first priority associated with the first controllable object is greater than a second priority associated with the second controllable object,

adjusting a first size of the first gesture recognition sub-area and a second size of the second gesture recognition sub-area based on the first priority and the second priority, respectively, such that the first size is larger than the second size,

detecting a gesture within the detection area;

determining one of the plurality of gesture recognition sub-areas in which the gesture is detected; and

executing an operation on one of the plurality of controllable objects based on the detected gesture in the determined one of the plurality of gesture recognition sub-areas.

Assignments (1)

Priority Claims (1)

Continuity (1)

Cited By (10)

US 12,202,655 US 12,249,034 US 12,277,654 US 12,321,509 US 12,406,454 US 12,474,788 US 12,505,626 US 12,547,369 US 12,554,333 US 12,591,297