IP Library Granted Patent US 11,539,935
Granted Patent B2
US 11,539,935 · App. 17/110,233 · Granted Dec 27, 2022

Videotelephony with parallax effect

Inventors: Andrew Garrod Bosworth (San Mateo, CA); Timo Juhani Ahonen (Belmont, CA); Brian Keith Cabral (San Jose, CA); Ryan Cairns (Los Altos Hills, CA); Andrea Colaco (Los Altos, CA); Jonathan Huang (Sunnyvale, CA); Michael Fredrick Cohen (Seattle, WA)
Assignee: Meta Platforms Technologies, LLC
H04N13/282H04N13/117H04N13/156H04N13/243H04N13/366H04N13/398H04N7/141H04N2013/0074
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,539,935
App. No.
17/110,233
Granted
Dec 27, 2022
Kind
B2
Abstract

In one embodiment, a computing system may receive, from a second computing system, video streams of a scene, the video streams including at least a first image and a second image that are simultaneously captured by a first camera and a second camera of the second computing system, respectively. The system may determine, using a sensor system, a viewpoint of a viewer with respect to a display region of a monoscopic display associated with the first computing system. The system may generate an output image of the scene by blending, according to blending proportions computed using the viewpoint of the viewer, corresponding portions of the first image and the second image. The system may display the output image in the display region of the monoscopic display.

Claims (42)

1. A method comprising, by a first computing system:

receiving, from a second computing system, video streams of a scene, the video streams comprising at least a first image and a second image that are simultaneously captured by a first camera and a second camera of the second computing system, respectively;

determining, using a sensor system, a view angle of a viewer with respect to a display region of a monoscopic display associated with the first computing system;

generating an output image of the scene by blending, according to blending proportions computed using the view angle of the viewer, at least a first portion of the first image and a second portion of the second image, wherein pixel values of the output image are determined based on (1) values of the blending proportions computed using the viewpoint view angle of the viewer and (2) corresponding pixel values of at least the first portion of the first image and the second portion of the second image, and wherein the values of the blending proportions are determined based on a comparison of the view angle of the viewer to a predetermined view angle range; and

displaying the output image in the display region of the monoscopic display.

2. The method of claim 1 , wherein the first portion of the first image and the second portion of the second image are identified by matching one or more first features captured in the first image to one or more second features captured in the second image, and wherein the first portion of the first image and the second portion of the second image are associated with a same feature.

3. The method of claim 1 , wherein a pixel value in the output image is determined based on a weighted average of the corresponding pixel values in the first portion of the first image and the second portion of the second image according to the blending proportions computed using the view angle of the viewer.

4. The method of claim 1 , wherein the first camera and the second camera of the second computing system are arranged at different positions, and wherein the first image and the second image of the scene are captured from different perspectives.

5. The method of claim 1 , wherein:

the video streams further comprise a third image captured simultaneously with the first image and the second image by a third camera associated with the second computing system,

the first camera, the second camera, and the third camera are arranged at three positions forming a two-dimensional plane, and

the output image of the scene is generated by blending, according to the blending proportions computed using the view angle of the viewer, the first portion of the first image, the second portion of the second image, and a third portion of the third image.

6. The method of claim 5 ,

wherein the view angle is a two-dimensional view angle determined based on a viewpoint of the viewer, and wherein the blending proportions are determined based at least on the two-dimensional view angle of the viewer.

7. The method of claim 6 , wherein the blending proportions are determined based on an angular proportion value of the two-dimensional view angle of the viewer with respect to boundary angles of a pre-determined conical volume corresponding to a viewable range for a head-motion parallax effect.

8. The method of claim 1 , wherein:

the video streams further comprise a third image of the scene captured by a third camera of the second computing system and a fourth image of the scene captured by a fourth camera of the second computing system, the third image and the fourth image being captured simultaneously with the first image and the second image;

the output image of the scene is generated based on a bilinear interpolation of the first image, the second image, the third image, and the fourth image of the scene according to a relative position of a viewpoint of the viewer with respect to the display region.

9. The method of claim 1 , further comprising:

receiving extrinsic camera properties of the first camera and the second camera, wherein the blending proportions are determined based on the extrinsic camera properties of the first camera and the second camera and the view angle of the viewer.

10. The method of claim 1 , further comprising:

determining an opposite view angle based on the view angle of the viewer, wherein the opposite view angle is symmetric to the view angle of the viewer with respect to an axis perpendicular to the display region;

in response to a user input switching a display mode of the first computing system, generating a subsequent output image based on the opposite view angle; and

displaying the subsequent output image on the monoscopic display to the viewer, wherein a character in the scene appears to follow the viewer when the view angle of the viewer changes.

11. The method of claim 1 , wherein the first camera and the second camera of the second computing system are arranged along an oblique direction with respect to the scene, wherein the view angle is a two-dimensional angle of the viewer determined based on a viewpoint of the viewer, and wherein the output image of the scene is generated based on the two-dimensional view angle of the viewer with respect to the display region.

12. The method of claim 1 , wherein the output image is displayed at a fixed position with respect to the display region when the view angle of the viewer changes with respect to the display region.

13. The method of claim 1 , wherein the output image is displayed at a dynamic position with respect to the display region when the view angle of the viewer changes with respect to the display region, and wherein the dynamic position for displaying the output image is determined based on the view angle of the viewer.

14. The method of claim 1 , wherein the output image is generated based on a virtual camera position configured by the viewer.

15. One or more computer-readable non-transitory storage media embodying software that is operable when executed by a first computing system to:

receive, from a second computing system, video streams of a scene, the video streams comprising at least a first image and a second image that are simultaneously captured by a first camera and a second camera of the second computing system, respectively;

determine, using a sensor system, a view angle of a viewer with respect to a display region of a monoscopic display associated with the first computing system;

generate an output image of the scene by blending, according to blending proportions computed using the view angle of the viewer, at least a first portion of the first image and a second portion of the second image, wherein pixel values of the output image are determined based on (1) values of the blending proportions computed using the view angle of the viewer and (2) corresponding pixel values of at least the first portion of the first image and the second portion of the second image, and wherein the values of the blending proportions are determined based on a comparison of the view angle of the viewer to a predetermined view angle range; and

display the output image in the display region of the monoscopic display.

16. The media of claim 15 , wherein the first portion of the first image and the second portion of the second image are identified by matching one or more first features captured in the first image to one or more second features captured in the second image, and wherein the first portion of the first image and the second portion of the second image are associated with a same feature.

17. The media of claim 15 , wherein a pixel value in the output image is determined based on a weighted average of corresponding pixel values in the first portion of the first image and the second portion of the second image according to the blending proportions computed using the view angle of the viewer.

18. A system comprising: one or more processors; and a memory coupled to the processors comprising instructions executable by the processors, the processors being operable when executing the instructions to:

receive, from a second computing system, video streams of a scene, the video streams comprising at least a first image and a second image that are simultaneously captured by a first camera and a second camera of the second computing system, respectively;

determine, using a sensor system, a view angle of a viewer with respect to a display region of a monoscopic display associated with the first computing system;

generate an output image of the scene by blending, according to blending proportions computed using the view angle of the viewer, at least a first portion of the first image and a second portion of the second image, wherein pixel values of the output image are determined based on (1) values of the blending proportions computed using the view angle of the viewer and (2) corresponding pixel values of at least the first portion of the first image and the second portion of the second image, and wherein the values of the blending proportions are determined based on a comparison of the view angle of the viewer to a predetermined view angle range; and

display the output image in the display region of the monoscopic display.

19. The system of claim 18 , wherein the first portion of the first image and the second portion the second image are identified by matching one or more first features captured in the first image to one or more second features captured in the second image, and wherein the first portion of the first image and the second portion of the second image are associated with a same feature.

20. The system of claim 18 , wherein a pixel value in the output image is determined based on a weighted average of corresponding pixel values in the first portion of the first image and the second portion of the second image according to the blending proportions computed using the view angle of the viewer.

Assignments (2)
CHANGE OF NAME Recorded Jul 6, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060591/0848 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 28, 2021
From: BOSWORTH, ANDREW GARROD; AHONEN, TIMO JUHANI; CABRAL, BRIAN KEITH; CAIRNS, RYAN; COLACO, ANDREA; HUANG, JONATHAN; COHEN, MICHAEL FREDRICK
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 055065/0513 →