IP Library Granted Patent US 10,198,866
Granted Patent B2
US 10,198,866 · App. 14/897,082 · Granted Feb 5, 2019

Head-mountable apparatus and systems

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,198,866
App. No.
14/897,082
Granted
Feb 5, 2019
Kind
B2
Abstract

A head mountable display (HMD) includes a camera operable to capture images of a peripheral and/or control device in use by a wearer of the HMD. A detector of the HMD is configured to detect occlusions in a captured image of the peripheral and/or control device. And an image renderer of the HMD is configured to render a virtual version of the peripheral and/or control device for display to the HMD wearer and to render a representation of a user's hand at a position of a detected occlusion.

Claims (40)

1. A head mountable display (HMD) system comprising:

a camera configured to capture images of a peripheral and/or control device in use by a wearer of the HMD;

a detector configured to detect an occlusion, by a hand of the wearer in a captured image, of the peripheral and/or control device; and

an image renderer configured to render a virtual version of the peripheral and/or control device for display to the HMD wearer and to render a representation of a user's hand at a position of the detected occlusion in the captured image.

2. An HMD system according to claim 1 , in which the representation of the user's hand is a virtual representation of the user's hand.

3. An HMD system according to claim 2 , in which:

the detector is configured to detect at least an angular orientation of the peripheral and/or control device from a shape of the peripheral and/or control device in a captured image; and

the image renderer is configured to render the virtual representation in dependence upon the detected angular orientation.

4. An HMD system according to claim 1 , in which:

the detector is configured to detect pixels corresponding to skin colours around a detected occlusion, and

the image renderer is configured to render parts of the captured image corresponding to the user's hand for display to the HMD wearer.

5. An HMD system according to claim 2 , in which the image renderer is configured to select a hand to be rendered based at least in part on either or both of user profile data indicating the user's dominant hand, and the position of the occlusion relative to the peripheral and/or control device.

6. An HMD system according to claim 1 , in which the HMD is configured to display computer game material.

7. An HMD system according to claim 1 , in which the detector is configured to detect one or both of markings and illumination on the peripheral and/or control device in the captured images.

8. An HMD system according to claim 1 , in which the detector is configured to detect false occlusions by comparing a detected occlusion with sensor data relating to the peripheral and/or control device.

9. An HMD system according to claim 1 , in which the detector is configured to detect false occlusions by detecting inconsistencies with control data from one or more other peripheral and/or control devices indicating the user touching the other peripheral and/or control devices.

10. An HMD system according to claim 1 , in which the detector is configured to detect false occlusions by detecting images of the peripheral and/or control device captured by another camera.

11. An HMD system according to claim 1 , in which the HMD system is configured to enable or disable haptic feedback for a given peripheral and/or control device in dependence upon whether an occlusion of the given peripheral and/or control device is detected.

12. The HMD system according to claim 2 , wherein the image renderer is configured to render the virtual representation of the user's hand by deciding which hand should be rendered based upon either or both of (a) an indication of which hand of the user is dominant, and (b) a current direction of movement of the peripheral and/or control device.

13. A computer games system, comprising:

a head mountable display (HMD) system including:

a camera configured to capture images of a peripheral and/or control device in use by a wearer of the HMD;

a detector configured to detect an occlusion, by a hand of the wearer in a captured image, of the peripheral and/or control device; and

an image renderer configured to render a virtual version of the peripheral and/or control device for display to the HMD wearer and to render a representation of a user's hand at a position of the detected occlusion in the captured image; and

one or more other devices selected from the list consisting of: a games console and a break-out box.

14. A computer games system according to claim 13 , in which one or more of the games console and the break-out box comprises a camera configured to detect images of the peripheral and/or control device.

15. A computer games system according to claim 13 , in which one or more of the games console and the break-out box comprises a light source configured to illuminate at least part of a rear aspect, with reference to a user of the HMD, of the peripheral and/or control device.

16. A computer games system according to claim 15 , in which the illuminator is an infra-red illuminator.

17. The computer games system according to claim 13 , wherein the image renderer is configured to render the representation of the user's hand by deciding which hand should be rendered based upon either or both of (a) an indication of which hand of the user is dominant, and (b) a current direction of movement of the peripheral and/or control device.

18. A method of operation of a head mountable display (HMD) comprising:

capturing images of a peripheral and/or control device in use by a wearer of the HMD;

detecting an occlusion, by a hand of the wearer in a captured image, of the peripheral and/or control device;

rendering, by one or more processors, a virtual version of the peripheral and/or control device for display to the HMD wearer; and

rendering, by the one or more processors, a representation of a user's hand at a position of the detected occlusion in the captured image.

19. The method of operation of the HMD according to claim 18 , wherein rendering the representation of the user's hand comprises deciding which hand should be rendered based upon either or both of (a) an indication of which hand of the user is dominant, and (b) a current direction of movement of the peripheral and/or control device.

20. A non-transitory computer-readable storage medium which stores computer instructions thereon, the instructions, when executed by one or more processors, causes the one or more processors to perform a method of operation of a head mountable display (HMD) comprising:

capturing images of a peripheral and/or control device in use by a wearer of the HMD;

detecting an occlusion, by a hand of the wearer in a captured image, of the peripheral and/or control device;

rendering, by one or more processors, a virtual version of the peripheral and/or control device for display to the HMD wearer; and

rendering, by the one or more processors, a representation of a user's hand at a position of the detected occlusion in the captured image.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 19, 2020
From: SONY INTERACTIVE ENTERTAINMENT EUROPE LIMITED
To: SONY INTERACTIVE ENTERTAINMENT INC.
Reel/Frame 052167/0398 →
CHANGE OF NAME Recorded Jul 14, 2017
From: SONY COMPUTER ENTERTAINMENT EUROPE LIMITED
To: SONY INTERACTIVE ENTERTAINMENT EUROPE LIMITED
Reel/Frame 043198/0110 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 10, 2015
From: BICKERSTAFF, IAN HENRY; BENSON, SIMON MARK; RAGHOEBARDAJAL, SHARWIN WINESH; MYATT, DARREN
To: SONY COMPUTER ENTERTAINMENT EUROPE LIMITED
Reel/Frame 037261/0245 →