Methods and apparatus for supporting content generation, transmission and/or playback
Methods and apparatus for supporting the capture of images of surfaces of an environment visible from a default viewing position and capturing images of surfaces not visible from the default viewing position, e.g., occluded surfaces, are described. Occluded and non-occluded image portions are packed into one or more frames and communicated to a playback device for use as textures which can be applied to a model of the environment where the images were captured. An environmental model includes a model of surfaces which are occluded from view from a default viewing position but which maybe viewed is the user shifts the user's viewing location. Occluded image content can be incorporated directly into a frame that also includes non-occluded image data or sent in frames of a separate, e.g., auxiliary content stream that is multiplexed with the main content stream which communicates image data corresponding to non-occluded environmental portions.
1. A method of operating an image processing system, the method comprising:
receiving a first image corresponding to a portion of an environment, said first image including a non-occluded image portion corresponding to a portion of the environment visible from a first location, wherein said first image is an image that was captured by a first camera of a stereoscopic camera pair;
generating a first frame including image content from said non-occluded image portion of said first image;
receiving an additional image of the environment including at least a first occluded image portion corresponding to a portion of the environment occluded from view from said first location, wherein said additional image is captured by an additional camera which is in addition to said stereoscopic camera pair;
generating an auxiliary frame including image content from said first occluded image portion of the additional image; and
storing said first frame and said auxiliary frame in a storage device or transmitting said first frame to another device.
2. The method of claim 1 , further comprising:
generating a first UV map indicating a mapping of portions of said first frame to segments of an environmental model which correspond to portions of said environment which are visible from said first location; and
generating an auxiliary UV map indicating a mapping of portions of said auxiliary frame to segments of an environmental model which correspond to portions of said environment which are not visible from said first location.
3. The method of claim 1 , wherein said auxiliary frame is smaller than said first frame and includes fewer pixels than said first frame.
4. The method of claim 1 , further comprising:
receiving a first request for content corresponding to a first program from a first playback device which supports the display of occluded image content; and
sending said first frame and the auxiliary frame to the first playback device in response to said first request for content.
5. The method of claim 4 , further comprising:
receiving a second request for content corresponding to the first program from a second playback device which does not support the display of occluded image content; and
sending said first frame to the second playback device in response to said second request for content without sending said auxiliary frame to the playback second device.
6. The method of claim 1 , wherein said additional camera is a different type of camera than said first camera.
7. The method of claim 1 , wherein the method includes storing said first frame and said auxiliary frame in a storage device, the method further comprising:
encoding said first frame prior to storage in said storage device; and
encoding said auxiliary frame prior to storage in said storage device.
8. A method of operating an image processing system, the method comprising:
generating an environmental model including segments corresponding to non-occluded surfaces of an environment and segments corresponding to occluded surfaces of said environment;
generating a first UV map indicating a mapping of portions of a first frame to segments of said environmental model which correspond to portions of said environment which are visible from a first location;
generating an auxiliary UV map indicating a mapping of portions of an auxiliary frame to segments of said environmental model which correspond to portions of said environment which are not visible from said first location;
receiving a first image corresponding to a portion of an environment, said first image including a non-occluded image portion corresponding to a portion of the environment visible from said first location;
generating said first frame including image content from said non-occluded image portion of said first image;
receiving an additional image of the environment including at least a first occluded image portion corresponding to a portion of the environment occluded from view from said first location;
generating said auxiliary frame including image content from said first occluded image portion of the additional image; and
storing said first frame and said auxiliary frame in a storage device or transmitting said first frame to another device.
9. The method of claim 8 , further comprising:
multiplexing said first frame and said auxiliary frame; and
transmitting a multiplexed content stream including said first frame in a primary content stream and said auxiliary frame in an auxiliary content stream.
10. The method of claim 9 , wherein multiplexing said first frame and said auxiliary frame includes incorporating said auxiliary frame in said multiplexed content stream before said first frame such that a device receiving said multiplexed content stream will receive said auxiliary frame before said first frame.
11. The method of claim 9 , further comprising:
receiving a second image corresponding to said portion of the environment, said second image including a second non-occluded image portion;
receiving a second additional image of the environment including at least a second occluded image portion; and
generating a second frame including image content from said second non-occluded image portion of said second image.
12. The method of claim 11 ,
wherein generating the auxiliary frame includes:
including image content from said second occluded image portion of the second additional image in said auxiliary frame with said first occluded image portion.
13. The method of claim 12 ,
wherein said auxiliary frame includes occluded image portions corresponding to M different frames in the primary content stream, M being a non-zero integer; and
wherein generating the auxiliary frame includes including image content from said second occluded image portion of the second additional image in said auxiliary frame.
14. The method of claim 13 , wherein said auxiliary frame is the same size as said first frame and includes the same number of pixels as said first frame.
15. A non-transitory computer readable medium for use in a system, said non-transitory computer readable medium including computer executable instructions which, when executed by a computer, control the system to:
receive a first image corresponding to a portion of an environment, said first image including a non-occluded image portion corresponding to a portion of the environment visible from a first location, wherein said first image is an image that was captured by a first camera of a stereoscopic camera pair;
generate a first frame including image content from said non-occluded image portion of said first image;
receive an additional image of the environment including at least a first occluded image portion corresponding to a portion of the environment occluded from view from said first location, wherein said additional image is captured by an additional camera which is in addition to said stereoscopic camera pair;
generate an auxiliary frame including image content from said first occluded image portion of the additional image; and
store said first frame and said auxiliary frame in a storage device or transmit said first frame to another device.
16. An image processing system, comprising:
a receiver configured to receive a first image corresponding to a portion of an environment, said first image including a non-occluded image portion corresponding to a portion of the environment visible from a first location;
a processor configured to generate a first frame including image content from said non-occluded image portion of said first image;
wherein said receiver is further configured to receive an additional image of the environment including at least a first occluded image portion corresponding to a portion of the environment occluded from view from said first location;
wherein said processor is further configured to generate an auxiliary frame including image content from said first occluded image portion of the additional image;
a memory for storing said first frame and said auxiliary frame or a transmitter configured to transmit both said first frame and said auxiliary frame to another device; and
wherein said processor is further configured to:
generate a first UV map indicating a mapping of portions of said first frame to segments of an environmental model which correspond to portions of said environment which are visible from said first location; and
generate an auxiliary UV map indicating a mapping of portions of said auxiliary frame to segments of an environmental model which correspond to portions of said environment which are not visible from said first location.
17. The image processing system of claim 16 , wherein said processor is further configured to generate an environmental model including segments corresponding to non-occluded surfaces of said environment and segments corresponding to occluded surfaces of said environment.
18. The image processing system of claim 17 , further comprising:
a multiplexer configured to multiplex said first frame and said auxiliary frame; and
wherein said transmitter is further configured to transmit a multiplexed content stream including said first frame in a primary content stream and said auxiliary frame in an auxiliary content stream.
19. The image processing system of claim 18 , further comprising:
wherein said multiplexer is configured, as part of being configured to multiplex said first frame and said auxiliary frame, to incorporate said auxiliary frame in said multiplexed content stream before said first frame such that a device receiving said multiplexed content stream will receive said auxiliary frame before said first frame.
20. The image processing system of claim 16 ,
wherein said receiver is further configured to receive a first request for content corresponding to a first program from a first playback device which supports the display of occluded image content;
wherein said transmitter is further configured to send said first frame and the auxiliary frame to the first playback device in response to said first request for content;
wherein said receiver is further configured to receive a second request for content corresponding to the first program from a second playback device which does not support the display of occluded image content; and
wherein said transmitter is further configured to send said first frame to the second playback device in response to said second request for content without sending said auxiliary frame to the playback second device.