IP Library Granted Patent US 10,757,399
Granted Patent B2
US 10,757,399 · App. 14/850,114 · Granted Aug 25, 2020

Stereo rendering system

Inventor: Andrew Ian Russell (San Jose, CA)
Assignee: GOOGLE LLC
H04N13/344G02B27/017G02B27/0172H04N13/117H04N13/133H04N13/15H04N13/366G02B2027/014G02B2027/0112G02B2027/0134G02B2027/0138
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,757,399
App. No.
14/850,114
Filed
Sep 10, 2015
Granted
Aug 25, 2020
Kind
B2
Art Unit
2425
USPC
348/53
Abstract

A method includes receiving an indication of a field of view associated with a three-dimensional (3D) image being displayed on a head mount display (HMD), receiving an indication of a depth of view associated with the 3D image being displayed on the HMD, selecting a first right eye image and a second right eye image based on the field of view, combining the first right eye image and the second right eye image based on the depth of view, selecting a first left eye image and a second left eye image based on the field of view, and combining the first left eye image and the second left eye image based on the depth of view.

Claims (57)

1. A method comprising:

receiving an indication of a field of view associated with a three-dimensional (3D) image being displayed on a head mount display (HMD);

receiving an indication of a depth of view as a focal depth detected on the HMD when displaying the 3D image being displayed on the HMD;

selecting a first right eye image as an image captured by a first camera positioned to the right of a right eye position and a second right eye image as an image captured by a second camera positioned to the left of the right eye position, the right eye position being based on the field of view;

combining the first right eye image and the second right eye image by overlaying the second right eye image on the first right eye image;

in response to receiving the indication of the depth of view, shifting the second right eye image horizontally with respect to the first right eye image until a portion of the second right eye image overlaps at least a portion of the first right eye image to match the depth of view;

selecting a first left eye image obtained from the left of a left eye position and a second left eye image obtained from the right of the left eye position, the left eye position being based on the field of view; and

combining the first left eye image and the second left eye image by overlaying the second left eye image on the first left eye image; and

in response to receiving the indication of the depth of view, shifting the second left eye image horizontally with respect to the first left eye image until a portion of the second left eye image overlaps at least a portion of the first left eye image to match the depth of view.

2. The method of claim 1 , wherein selecting of the first left eye image and of the second left eye image include:

determining, a left eye position of a user of the HMD;

selecting the first left eye image as an image taken by a first camera positioned to the right of the left eye position; and

selecting the second left eye image as an image taken by a second camera positioned to the left of the left eye position.

3. The method of claim 1 , further comprising:

selecting additional left eye images and additional right eye images;

overlaying at least two of the additional left eye images; and

in response to receiving the indication of the depth of view, shifting the at least two additional left eye images until a portion of each of the at least two left eye images include an overlapped portion that match the depth of view;

overlaying at least two of the additional right eye images; and

in response to receiving the indication of the depth of view, shifting the at least two additional right eye images until a portion of each of the at least two right eye images include an overlapped portion that match the depth of view.

4. The method of claim 1 , wherein the combining of the first right eye image and the second right eye image includes shifting the first right eye image with respect to the second right eye image until at least a portion of a combined image, based on the depth of view, is undistorted.

5. The method of claim 1 , wherein the combining of the first right eye image and the second right eye image includes shifting both the first right eye image and the second right eye image toward the center of the field of view until a portion of a combined image, based on the depth of view, is undistorted.

6. The method of claim 1 , wherein the combining of the first right eye image and the second right eye image includes color merging such that a portion of the combined image has a same color palette as a corresponding portion of at least one of the first right eye image and the second right eye image.

7. The method of claim 1 , wherein the combining of the first right eye image and the second right eye image includes color merging using a color offset based on a weighted offset associated with a camera distance from the center of the field of view.

8. The method of claim 1 , wherein:

combining and shifting of the first right eye image and the second right eye image and combining and shifting the first left eye image and the second left eye image is performed without optical flow interpolation.

9. The method of claim 1 , wherein the combining of the first left eye image and the second left eye image includes shifting the first left eye image with respect to the second left eye image until a portion of a combined image, based on the depth of view, is sharp.

10. The method of claim 1 , wherein the combining of the first left eye image and the second left eye image includes shifting both the first left eye image and the second left eye image toward the center of the field of view until a portion of a combined image, based on the depth of view, is sharp.

11. The method of claim 1 , wherein the combining of the first left eye image and the second left eye image includes color merging the combined image such that a portion of the combined image has a same color palette as a corresponding portion of at least one of the first left eye image and the second left eye image.

12. The method of claim 1 , wherein the combining of the first left eye image and the second left eye image includes color merging the combined image using a color offset based on a weighted offset associated with a camera distance from the center of the field of view.

13. A method comprising:

streaming a three-dimensional (3D) video to a head mount display (HMD), each frame of the 3D video including a left eye image and a right eye image; and

generating a subsequent frame includes:

determining a field of view associated with the 3D video;

determining a depth of view as a focal depth detected on the HMD when displaying the 3D video;

selecting a first right eye image obtained from the left of a right eye position and a second right eye image, obtained from the right of the right eye position the right eye position being based on the field of view;

overlaying the second right eye image on the first right eye image;

in response to receiving the indication of the depth of view, shifting the second right eye image horizontally with respect to the first right eye image until a portion of the second right eye image overlaps at least a portion of the first right eye image to match the depth of view;

selecting a first left eye image as an image captured by a first camera positioned to the right of a left eye position and a second left eye image as an image captured by a second camera positioned to the left of the left eye position, the left eye position being based on the field of view;

overlaying the second left eye image on the first left eye image; and

in response to receiving the indication of the depth of view, shifting the second left eye image horizontally with respect to the first left eye image until a portion of the second left eye image overlaps at least a portion of the first left eye image to match the depth of view.

14. The method of claim 13 , wherein selecting of the first right eye image and of the second right eye image includes:

determining a right eye position of a user of the HMD, selecting the first right eye image as an image taken by a first camera positioned to the right of the right eye position, and

selecting the second right eye image as an image taken by a second camera positioned to the left of the right eye position.

15. The method of claim 13 , wherein the first right eye image, the second right eye image, the first left eye image and the second left eye image are selected from a plurality of images captured at the same moment in time.

16. The method of claim 13 , wherein the combining of the first right eye image and the second right eye image includes shifting the first right eye image with respect to the second right eye image until a portion of a combined image, based on the depth of view, is undistorted, and the combining of the first left eye image and the second left eye image includes shifting the first left eye image with respect to the second left eye image until a portion of a combined image, based on the depth of view, is undistorted.

17. The method of claim 13 , wherein the combining of the first right eye image and the second right eye image includes shifting both the first right eye image and the second right eye image toward the center of the field of view until a portion of a combined image, based on the depth of view, is undistorted, and

the combining of the first left eye image and the second left eye image includes shifting both the first left eye image and the second left eye image toward the center of the field of view until a portion of a combined image, based on the depth of view, is undistorted.

18. The method of claim 13 , wherein the combining of the first left eye image and the second left eye image includes color merging the combined image.

19. A non-transitory computer-readable storage medium having stored thereon computer executable program code which, when executed on a computer system, causes the computer system to perform steps comprising:

receiving an indication of a field of view associated with a three-dimensional (3D) image being displayed on a head mount display (HMD);

receiving an indication of a depth of view as a focal depth of a user of the HMD viewing the 3D image being displayed on the HMD;

selecting a first right eye image as an image captured by a first camera positioned to the right of a right eye position and a second right eye image as an image captured by a second camera positioned to the left of the right eye position, the right eye position being based on the field of view;

overlaying the second right eye image on the first right eye image;

in response to receiving the indication of the depth of view, shifting the second right eye image horizontally with respect to the first right eye image until a portion of the second right eye image overlaps at least a portion of the first right eye image to match the depth of view;

selecting a first left eye image obtained from the left of a left eye position and a second left eye image obtained from the right of the left eye position, the left eye position being based on the field of view;

overlaying the second left eye image on the first left eye image; and

in response to receiving the indication of the depth of view, shifting the second left eye image horizontally with respect to the first left eye image until a portion of the second left eye image overlaps at least a portion of the first left eye image to match the depth of view.

Assignments (2)
CHANGE OF NAME Recorded Oct 5, 2017
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 044129/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 10, 2015
From: RUSSELL, ANDREW IAN
To: GOOGLE INC.
Reel/Frame 036534/0687 →
Continuity (1)
Related Publication 20170078651A1 · Mar 16, 2017
Cited By (1)
US 12,535,880