IP Library Granted Patent US 10,397,543
Granted Patent B2
US 10,397,543 · App. 14/845,202 · Granted Aug 27, 2019

Methods and apparatus for capturing, streaming and/or playing back content

Inventors: David Cole (Laguna Beach, CA); Alan McKay Moss (Laguna Beach, CA); Hector M Medina (Laguna Beach, CA)
Assignee: NextVR Inc.
H04N13/172H04N13/117H04N13/139H04N13/161H04N13/189H04N13/194H04N13/239H04N13/275H04N13/366H04N13/398H04N19/597H04N13/246H04N13/344
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,397,543
App. No.
14/845,202
Granted
Aug 27, 2019
Kind
B2
Abstract

Methods and apparatus for streaming or playing back stereoscopic content are described. Camera dependent correction information is communicated to a playback device and applied in the playback device to compensate for distortions introduced by the lenses of individual cameras. By performing lens dependent distortion compensation in the playback device edges which might be lost if correction were performed prior to encoding are preserved. Distortion correction information maybe in the form of UV map correction information. The correction information may indicate changes to be made to information in a UV map, e.g., at rendering time, to compensate for distortions specific to an individual camera. Different sets of correction information maybe communicated and used for different cameras of a stereoscopic pair which provide images that are rendered using the same UV map. The communicated correction information is sometimes called a correction mesh since it is used to correct mesh related information.

Claims (61)

1. A method of providing stereoscopic content, the method comprising:

storing, in a content delivery system,

i) first stereoscopic camera pair correction information corresponding to a first stereoscopic camera pair, said correction information corresponding to the first stereoscopic camera pair including:

first correction information for a first camera of the first stereoscopic camera pair, said first correction information being first camera dependent mesh correction information, indicating corrections to be made to node positions in a UV map when rendering an image captured by said first camera; and second correction information for a second camera of the first stereoscopic camera pair, said second correction information being second camera dependent mesh correction information, indicating corrections to be made to node positions in the UV map when rendering an image captured by said second camera;

ii) second stereoscopic camera pair correction information corresponding to a second stereoscopic camera pair, said second stereoscopic camera pair including a third camera and a fourth camera, said correction information corresponding to the second stereoscopic camera pair including third correction information for images captured by the third camera and fourth correction information for images captured by the fourth camera; and

iii) said UV map;

operating the content delivery system to provide said UV map, said first correction information corresponding to the first stereoscopic camera pair and said second correction information corresponding to the second stereoscopic camera pair to a playback device prior to detecting a network controlled switch from streaming content from the first stereoscopic camera pair to streaming content corresponding to the second stereoscopic camera pair;

detecting a network controlled switch from streaming content from the first stereoscopic camera pair to streaming content second stereoscopic pair; and

in response to detecting the network controlled switch from streaming content from said first stereoscopic camera pair to said second stereoscopic pair, operating the content delivery system to indicate to the playback device that the second stereoscopic camera pair correction information should be used for rendering operations.

2. The method of claim 1 ,

wherein said first stereoscopic camera pair corresponds to a first direction, the method further comprising:

operating said content delivery system to transmit a stereoscopic content stream including encoded images generated from image content captured by said first and second cameras.

3. The method of claim 2 , further comprising:

transmitting to said playback device an environmental mesh model to be used in rendering image content using said UV map.

4. The method of claim 3 ,

wherein said first camera of the first stereoscopic camera pair captures left eye images and said second camera of the first stereoscopic camera pair captures right eye images;

wherein said first correction information includes information generated based on measurement of one or more optical characteristics of a first lens of said first camera of the first stereoscopic camera pair; and

wherein said second correction information includes information generated based on measurement of one or more optical characteristic of a second lens of said second camera of the first stereoscopic camera pair.

5. The method of claim 4 wherein said UV map maps portions of images captured by said first stereoscopic camera pair to a portion of said environmental mesh model as part of an image rendering operation.

6. The method of claim 5 , wherein the first correction information includes information identifying a subset of nodes in the UV map and provides node position correction information for the subset of nodes for which corrections are to be performed, said subset of nodes being less than all the nodes in the UV map.

7. The method of claim 6 , wherein the second correction information includes information identifying a subset of nodes in the UV map and provides node position correction information for the subset of nodes for which corrections are to be performed, said subset of nodes being less than all the nodes in the UV map.

8. The method of claim 2 , wherein said stereoscopic content stream communicates encoded image content including encoded left and right eye images captured by said first and second cameras of said first stereoscopic camera pair.

9. The method of claim 8 , further comprising:

operating the content delivery system to provide third correction mesh information and fourth correction mesh information to the playback device, said third correction mesh information being for use in rendering image content captured by a first camera of a second stereoscopic camera pair, said fourth correction mesh information being for use in rendering image content captured by a second camera of the second stereoscopic camera pair.

10. The method of claim 9 , wherein said first and second stereoscopic camera pairs correspond to a forward viewing direction but different locations at an area or event location where content is being captured for streaming.

11. The method of claim 9 , further comprising:

indicating to the playback device that said third and fourth correction mesh information should be used when content captured by said second camera pair is streamed to the playback device instead of content from said first camera pair.

12. The method of claim 11 , wherein said step of indicating to the playback device that said third and fourth correction mesh information should be used is performed in response to: a user control change from receiving a first content stream including content from said first stereoscopic camera pair to receiving a second content stream including encoded content from the second stereoscopic camera pair.

13. The method of claim 12 , further comprising:

storing fifth correction mesh information for a first camera of a third stereoscopic camera pair;

storing sixth correction mesh information for a second camera of the third stereoscopic camera pair; and

operating the content delivery system to provide said fifth and sixth correction mesh information to the playback device, said fifth correction mesh information being for use in rendering image content captured by said first camera of said third stereoscopic camera pair, said sixth correction mesh information being for use in rendering image content captured by said second camera of the third stereoscopic camera pair.

14. A content delivery system, comprising:

a memory including:

i) first stereoscopic camera pair correction information corresponding to a first stereoscopic camera pair, said correction information corresponding to the first stereoscopic camera pair including:

first correction information for a first camera of the first stereoscopic camera pair, said first correction information being first camera dependent mesh correction information, indicating corrections to be made to node positions in a UV map when rendering an image captured by said first camera; and second correction information for a second camera of the first stereoscopic camera pair, said second correction information being second camera dependent mesh correction information, indicating corrections to be made to node positions in the UV map when rendering an image captured by said second camera;

ii) second stereoscopic camera pair correction information corresponding to a second stereoscopic camera pair, said second stereoscopic camera pair including a third camera and a fourth camera, said correction information corresponding to the second stereoscopic camera pair including third correction information for images captured by the third camera and fourth correction information for images captured by the fourth camera; and

iii) said UV map; and

a processor configured to:

control said system to provide, to a playback device, said UV map, said first correction information corresponding to the first stereoscopic camera pair and said second correction information corresponding to the second stereoscopic camera pair, prior to detecting a network controlled switch from streaming content from the first stereoscopic camera pair to streaming content corresponding to the second stereoscopic camera pair;

detect a network controlled switch from streaming content from the first stereoscopic camera pair to streaming content corresponding to second stereoscopic pair; and

indicate, in response to detecting the network controlled switch from streaming content from said first stereoscopic camera pair to said streaming content corresponding to second stereoscopic pair, to the playback device that the second stereoscopic camera pair correction information should be used for rendering operations.

15. The content delivery system of claim 14 , wherein said first stereoscopic camera pair corresponds to a first direction;

wherein said processor is further configured to control said system to transmit a stereoscopic content stream including encoded images generated from image content captured by said first and second cameras; and

wherein said processor is further configured to transmit to said playback device an environmental mesh model to be used in rendering image content.

16. The content delivery system of claim 15 ,

wherein said first camera of the first stereoscopic camera pair captures left eye images and said second camera of the first stereoscopic camera pair captures right eye images;

wherein said first correction information includes information generated based on measurement of one or more optical characteristics of a first lens of said first camera of the first stereoscopic camera pair; and

wherein said second correction information includes information generated based on measurement of one or more optical characteristic of a second lens of said second camera of the first stereoscopic camera pair.

17. The content delivery system of claim 16 , wherein said processor is further configured to transmit to said playback device a first UV map to be used for mapping portions of images captured by said first stereoscopic camera pair to a portion of said environmental mesh model as part of an image rendering operation.

18. The content delivery system of claim 14 , wherein said processor is further configured to control said system to provide third correction mesh information and fourth correction mesh information to the playback device, said third correction mesh information being for use in rendering image content captured by a first camera of a second stereoscopic camera pair, said fourth correction mesh information being for use in rendering image content captured by a second camera of the second stereoscopic camera pair.

19. The content delivery system of claim 18 , wherein said processor is configured to indicate to the playback device that said third and fourth correction mesh information should be in response to said system detecting: i) a network controlled switch from streaming content from said first stereoscopic camera pair to said second stereoscopic pair or ii) a user controlled change from receiving a first content stream including content from said first stereoscopic camera pair to receiving a second content stream including encoded content from the second stereoscopic camera pair.

20. A non-transitory computer readable medium including processor executable instructions which when executed by a processor control a content delivery system to:

store, in the content delivery system,

i) first stereoscopic camera pair correction information corresponding to a first stereoscopic camera pair, said correction information corresponding to the first stereoscopic camera pair including:

first correction information for a first camera of the first stereoscopic camera pair, said first correction information being first camera dependent mesh correction information, indicating corrections to be made to node positions in a UV map when rendering an image captured by said first camera; and second correction information for a second camera of the first stereoscopic camera pair, said second correction information being second camera dependent mesh correction information, indicating corrections to be made to node positions in the UV map when rendering an image captured by said second camera;

ii) second stereoscopic camera pair correction information corresponding to a second stereoscopic camera pair, said second stereoscopic camera pair including a third camera and a fourth camera, said correction information corresponding to the second stereoscopic camera pair including third correction information for images captured by the third camera and fourth correction information for images captured by the fourth camera; and

iii) said UV map;

provide said UV map, said first correction information corresponding to the first stereoscopic camera pair and said second correction information corresponding to the second stereoscopic camera pair to a playback device prior to detecting a network controlled switch from streaming content from the first stereoscopic camera pair to streaming content corresponding to the second stereoscopic camera pair;

detect a network controlled switch from streaming content from the first stereoscopic camera pair to streaming content corresponding to the second stereoscopic pair; and

indicate, in response to detecting the network controlled switch from streaming content from said first stereoscopic camera pair to streaming content corresponding to the said second stereoscopic pair, to the playback device that the second stereoscopic camera pair correction information should be used for rendering operations.

Assignments (5)
NUNC PRO TUNC ASSIGNMENT Recorded Jul 7, 2020
From: NEXTVR INC.
To: NEVERMIND CAPITAL LLC
Reel/Frame 053139/0942 →
RELEASE OF SECURITY INTEREST Recorded May 13, 2020
From: VCAP VENTURES (BVI) LIMITED
To: NEXTVR, INC.
Reel/Frame 052657/0091 →
CHANGE OF ADDRESS Recorded Apr 9, 2019
From: NEXTVR INC.
To: NEXTVR INC.
Reel/Frame 048836/0085 →
SECURITY INTEREST Recorded Mar 29, 2019
From: NEXTVR INC.
To: VCAP VENTURES (BVI) LIMITED
Reel/Frame 048747/0128 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 17, 2016
From: COLE, DAVID; MOSS, ALAN MCKAY; MEDINA, HECTOR M
To: NEXTVR INC.
Reel/Frame 038016/0320 →
Continuity (2)
Provisional Application 62045004 · Sep 3, 2014
Related Publication 20160065946A1 · Mar 3, 2016