IP Library › Granted Patent US 11,272,150
Granted Patent B2
US 11,272,150 · App. 16/917,333 · Granted Mar 8, 2022

Methods and apparatus for supporting content generation, transmission and/or playback

Inventors: David Cole (Aliso Viejo, CA); Alan McKay Moss (Laguna Beach, CA)
Assignee: Nevermind Capital LLC
H04N13/111G06F3/011G06F3/012G06F3/04815G06T17/05G06T17/20G06T19/20G09G3/003G09G5/001G09G5/003G09G5/006H04N13/139H04N13/161H04N13/189H04N13/194H04N13/204H04N13/232H04N13/239H04N13/271H04N13/279H04N13/282H04N13/344H04N19/44H04N21/4345G06K9/00362G06T2207/30244G09G2352/00G09G2360/02G09G2360/125G09G2370/02
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,272,150
App. No.
16/917,333
Granted
Mar 8, 2022
Kind
B2
Abstract

Methods and apparatus for supporting the capture of images of surfaces of an environment visible from a default viewing position and capturing images of surfaces not visible from the default viewing position, e.g., occluded surfaces, are described. Occluded and non-occluded image portions are packed into one or more frames and communicated to a playback device for use as textures which can be applied to a model of the environment where the images were captured. An environmental model includes a model of surfaces which are occluded from view from a default viewing position but which may be viewed is the user shifts the user's viewing location. Occluded image content can be incorporated directly into a frame that also includes non-occluded image data or sent in frames of a separate, e.g., auxiliary content stream that is multiplexed with the main content stream which communicates image data corresponding to non-occluded environmental portions.

Claims (65)

1. A method of operating a system, comprising:

receiving, from a playback device, a first request for content;

determining whether the playback device is configured to enable multiple viewing positions;

in accordance with the first request for content and a determination that the playback device is configured to enable multiple viewing positions, determining whether the playback device is configured to demultiplex at least two data streams; and

in accordance with a determination that the playback device is not configured to demultiplex at least two data streams, transmitting a first frame to the playback device, wherein the first frame comprises non-occluded image data and occluded image data,

wherein the non-occluded image data corresponds to a first portion of an environment visible from a first viewing position, and

wherein the occluded image data corresponds to a second portion of the environment not visible from the first viewing position and visible from a second viewing position.

2. The method of claim 1 , further comprising:

receiving, from the playback device, a second request for content; and

in accordance with the second request and a determination that the playback device is not configured to enable multiple viewing positions, transmitting a second frame to the playback device comprising the non-occluded image data.

3. The method of claim 2 , wherein the first and second requests for content comprise data indicative of whether the playback device is configured to enable multiple viewing positions.

4. The method of claim 2 , wherein the non-occluded image data comprises non-occluded image content, a first UV map, and a first environmental model.

5. The method of claim 4 , wherein the occluded image data comprises occluded image content, a second UV map, and a second environmental model.

6. The method of claim 5 , further comprising:

receiving, from a second playback device, a third request for content;

determining whether the second playback device is configured to demultiplex at least two data streams; and

in accordance with a determination that the second playback device is configured to demultiplex at least two data streams, transmit a primary data stream and an auxiliary data stream to the second playback device, wherein the primary data stream comprises the second frame, wherein the auxiliary data stream comprises a third frame, and wherein the third frame comprises the occluded image data.

7. The method of claim 6 , wherein the occluded image data comprises occluded image content, a third UV map, and a third environmental model.

8. The method of claim 7 , wherein:

the first environmental model corresponds to the first portion of the environment visible from the first viewing position;

the first UV map is indicative of a mapping of the non-occluded image content to the first environmental model;

the second environmental model comprises a first segment corresponding to the first portion of the environment and a second segment corresponding to the second portion of the environment that is not visible from the first viewing position and visible from the second viewing position;

the second UV map is indicative of a mapping of the non-occluded image content to the first segment and the occluded image content to the second segment;

the third environmental model corresponds to the second portion of the environment; and

the third UV map is indicative of a mapping of the occluded image content to the third environmental model.

9. The method of claim 8 , further comprising:

receiving a first image and a second image, wherein the first image is captured by a first camera and comprises the non-occluded image content, wherein the second image is captured by a second camera and comprises the occluded image content; and

generating the first frame, the second frame, and the third frame based on the first and second images.

10. The method of claim 9 , further comprising generating the first, second, and third UV maps.

11. The method of claim 8 , further comprising:

receiving a depth map of an environment; and

generating the first, second, and third environmental models based on the depth map.

12. A non-transitory computer readable medium comprising computer code, executable by one or more processors to:

receive, from a playback device, a request for content;

determine whether the playback device is configured to enable multiple viewing positions;

in accordance with a determination that the playback device is not configured to enable multiple viewing positions, transmit a first frame to the playback device, wherein the first frame comprises non-occluded image data corresponding to a first portion of an environment visible from a first viewing position;

in accordance with a determination that the playback device is configured to enable multiple viewing positions, determine whether the playback device is configured to demultiplex at least two data streams;

in accordance with a determination that the playback device is not configured to demultiplex at least two data streams, transmit a second frame to the playback device, wherein the second frame comprises the non-occluded image data and occluded image data corresponding to a second portion of the environment not visible from the first viewing position and visible from a second viewing position; and

in accordance with a determination that the playback device is configured to demultiplex at least two data streams, transmit a primary data stream and an auxiliary data stream to the playback device, wherein the primary data stream comprises the first frame, wherein the auxiliary data stream comprises a third frame, and wherein the third frame comprises the occluded image data.

13. The non-transitory computer readable medium of claim 12 , wherein the request for content comprises data indicative of whether the playback device is configured to enable multiple viewing positions.

14. The non-transitory computer readable medium of claim 12 , wherein the first frame further comprises a first UV map and a first environmental model.

15. The non-transitory computer readable medium of claim 14 , wherein:

the first environmental model corresponds to a first portion of an environment visible from the first viewing position; and

the first UV map is indicative of a mapping of non-occluded image content to the first environmental model.

16. The non-transitory computer readable medium of claim 12 , wherein the second frame further comprises a second UV map and a second environmental model.

17. The non-transitory computer readable medium of claim 16 , wherein:

the second environmental model comprises a first segment corresponding to a first portion of an environment visible from the first viewing position and a second segment corresponding to a second portion of the environment that is not visible from the first viewing position and visible from the second viewing position; and

the second UV map is indicative of a mapping of non-occluded image content to the first segment and occluded image content to the second segment.

18. The non-transitory computer readable medium of claim 12 , wherein the third frame further comprises a third UV map and a third environmental model.

19. The non-transitory computer readable medium of claim 18 , wherein:

the third environmental model corresponds to a second portion of an environment that is not visible from the first viewing position and visible from the second viewing position; and

the third UV map is indicative of a mapping of occluded image content to the third environmental model.

20. A system, comprising:

one or more processors; and

one or more non-transitory computer readable media comprising computer readable code executable by the one or more processors to:

receive, from a playback device, a request for content;

determine whether the playback device is configured to enable multiple viewing positions;

in accordance with a determination that the playback device is not configured to enable multiple viewing positions, transmit a first frame to the playback device, wherein the first frame comprises non-occluded image data corresponding to a first portion of an environment visible from a first viewing position;

in accordance with a determination that the playback device is configured to enable multiple viewing positions, determine whether the playback device is configured to demultiplex at least two data streams;

in accordance with a determination that the playback device is not configured to demultiplex at least two data streams, transmit a second frame to the playback device, wherein the second frame comprises the non-occluded image data and occluded image data corresponding to a second portion of the environment not visible from the first viewing position and visible from a second viewing position; and

in accordance with a determination that the playback device is configured to demultiplex at least two data streams, transmit a primary data stream and an auxiliary data stream to the playback device, wherein the primary data stream comprises the first frame, wherein the auxiliary data stream comprises a third frame, and wherein the third frame comprises the occluded image data.

21. The system of claim 20 , wherein the request for content comprises data indicative of whether the playback device is configured to enable multiple viewing positions.

22. The system of claim 20 , wherein the first frame further comprises a first UV map and a first environmental model.

23. The system of claim 20 , wherein the second frame further comprises a second UV map and a second environmental model.

24. The system of claim 20 , wherein the third frame further comprises a third UV map and a third environmental model.

Continuity (6)
Continuation 16040325 · Jul 19, 2018
Continuation 15058125 · Mar 1, 2016
Provisional Application 62127215 · Mar 2, 2015
Provisional Application 62126701 · Mar 1, 2015
Provisional Application 62126709 · Mar 1, 2015
Related Publication 20200404236A1 · Dec 24, 2020