IP Library Granted Patent US 9,852,549
Granted Patent B2
US 9,852,549 · App. 15/007,701 · Granted Dec 26, 2017

Image processing

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,852,549
App. No.
15/007,701
Granted
Dec 26, 2017
Kind
B2
Abstract

An image processing method includes capturing an image of the head of a user of a head mountable display device. The position of the head mountable display device is detected in the image captured by a camera. A region of the user's face that is occluded by the head mountable display device is identified. The portion of the captured image corresponding to the head mountable display device is at least partially replaced with a corresponding portion of a 3D facial model, to provide a modified version of the captured image.

Claims (37)

1. An image processing method comprising:

capturing, by a camera, an image of a head of a user of a head mountable display device;

detecting, by a detector device, a position of the head mountable display device in the image captured by the camera;

identifying, by a processor device, a region of the user's face that is occluded by the head mountable display device; and

the processor device at least partially replacing a region of the captured image corresponding to the head mountable display device with a corresponding portion of a 3D facial model to provide a modified image.

2. The image processing method of claim 1 , comprising deriving the 3D facial model from captured images of the user's face.

3. The image processing method of claim 1 , in which the 3D facial model is a model of a face other than the user's face.

4. The image processing method of claim 1 , wherein the corresponding portion comprises a superposition of a partly transparent image of the head mountable display device and the 3D facial model.

5. The image processing method of claim 1 , wherein the 3D facial model comprises a mapping of interactions of facial muscles.

6. The image processing method of claim 1 , wherein the 3D facial model includes a representation of portions of a given face other than those portions directly overlaid by the head mountable display in use.

7. The image processing method of claim 1 , in which the identifying step comprises detecting an orientation of the head mountable display device and detecting occlusion of the user's face from a detected orientation of the head mountable display device.

8. The image processing method of claim 1 , in which the identifying step comprises comparing a captured image of the user's face with a reference image.

9. The image processing method of claim 1 , comprising rendering motion of the user's eyes in the modified image.

10. The image processing method of claim 9 , comprising detecting the eye motion using one or more cameras mounted on the head mountable display device.

11. The image processing method of claim 9 , comprising generating the eye motion by a computer.

12. The image processing method of claim 1 , comprising:

detecting facial muscle motion; and

changing generation of the modified image in accordance with the detected facial muscle motion.

13. The image processing method of claim 12 , in which the step of detecting facial muscle motion comprises estimating the facial muscle motion from captured images.

14. The image processing method of claim 12 , in which the step of detecting facial muscle motion comprises detecting the facial muscle motion using electromyography.

15. A machine-readable non-transitory storage medium which stores computer readable instructions thereon that, when executed by a computer, cause the computer to perform an image processing method comprising:

capturing an image of a head of a user of a head mountable display device;

detecting a position of the head mountable display device in the image captured by a camera;

identifying a region of the user's face that is occluded by the head mountable display device; and

at least partially replacing a region of the captured image corresponding to the head mountable display device with a corresponding portion of a 3D facial model, to provide a modified image.

16. An image processing apparatus operable to process an image, comprising:

an image capturing device configured to capture an image of a head of a user of a head mountable display device;

a detector configured to detect a position of the head mountable display device in the image captured by the image capturing device; and

a processor configured to identify a region of the user's face that is occluded by the head mountable display device, the processor being further configured to replace, at least partially, a portion of the captured image corresponding to the head mountable display device with a corresponding portion of a 3D facial model, to provide a modified image.

17. A communications system comprising:

first and second head mountable displays;

one or more cameras configured to capture images of the first and second head mountable displays in use; and

an image processing apparatus comprising:

a detector configured to detect positions of the first and second head mountable displays in the images captured by the one or more cameras; and

a processor configured to:

replace, at least partially, a portion of a captured image corresponding to the first head mountable display device with a corresponding portion of a 3D facial model, to provide a first modified image for display by the second head mountable display, and

replace, at least partially, a portion of a captured image corresponding to the second head mountable display with a corresponding portion of the 3D facial model, to provide a second modified image for display by the first head mountable display.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 19, 2020
From: SONY INTERACTIVE ENTERTAINMENT EUROPE LIMITED
To: SONY INTERACTIVE ENTERTAINMENT INC.
Reel/Frame 052167/0398 →
CHANGE OF NAME Recorded Jul 14, 2017
From: SONY COMPUTER ENTERTAINMENT EUROPE LIMITED
To: SONY INTERACTIVE ENTERTAINMENT EUROPE LIMITED
Reel/Frame 043198/0110 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 8, 2016
From: RAGHOEBARDAJAL, SHARWIN WINESH; BENSON, SIMON MARK; ASHFORTH, JEREMY DAVID; BICKERSTAFF, IAN HENRY
To: SONY COMPUTER ENTERTAINMENT EUROPE LIMITED
Reel/Frame 038229/0771 →