Methods and apparatus for supporting content generation, transmission and/or playback
Methods and apparatus for supporting the capture of images of surfaces of an environment visible from a default viewing position and capturing images of surfaces not visible from the default viewing position, e.g., occluded surfaces, are described. Occluded and non-occluded image portions are packed into one or more frames and communicated to a playback device for use as textures which can be applied to a model of the environment where the images were captured. An environmental model includes a model of surfaces which are occluded from view from a default viewing position but which maybe viewed is the user shifts the user's viewing location. Occluded image content can be incorporated directly into a frame that also includes non-occluded image data or sent in frames of a separate, e.g., auxiliary content stream that is multiplexed with the main content stream which communicates image data corresponding to non-occluded environmental portions.
1. A method of operating an image processing system, the method comprising:
receiving a first image corresponding to a portion of an environment, said first image including a non-occluded image portion corresponding to a portion of the environment visible from a first location;
receiving an additional image of the environment including at least a first occluded image portion corresponding to a portion of the environment occluded from view from said first location;
generating a first frame including image content from said non-occluded image portion of said first image and image content from said first occluded image portion of the additional image; and
storing said first frame in a storage device or transmitting said first frame to another device.
2. The method of claim 1 , further comprising:
generating a UV map to be used for mapping portions of said first frame to segments of an environmental model.
3. The method of claim 1 , further comprising:
encoding said first frame prior to storing said first frame in the storage device; and
generating an environmental model including segments corresponding to non-occluded surfaces of said environment and segments corresponding to occluded surfaces of said environment.
4. The method of claim 3 , wherein said UV map maps image content in said first frame corresponding to said non-occluded image portion of said first frame to a first segment of said environmental model which corresponds to a portion of said environment which is visible from said first location.
5. The method of claim 4 , wherein said UV map further maps image content in said first frame corresponding to said first occluded image portion to a second segment of said environmental model which corresponds to a portion of said environment which is not visible from said first location.
6. The method of claim 1 ,
wherein receiving a first image includes receiving said first image from a first camera in said environment; and
wherein receiving an additional image of the environment includes receiving said additional image from an additional camera at a location in said environment offset from a location at which said first camera is positioned in said environment.
7. The method of claim 6 , wherein said first camera is a camera of a stereoscopic camera pair including said first camera and a second camera, said stereoscopic camera pair being positioned at said first location, the method further comprising:
receiving a second image corresponding to a portion of an environment from the second camera, said second image including a second non-occluded image portion corresponding to the portion of the environment visible from the first location; and
wherein said additional camera is a camera which is in addition to said first and second cameras of the stereoscopic camera pair used to capture non-occluded image content.
8. The method of claim 7 , wherein said additional camera is a camera of a different type than said first and second cameras of said stereoscopic camera pair.
9. The method of claim 7 , wherein said additional camera is a lower resolution camera than said first and second cameras.
10. The method of claim 6 , wherein generating a first frame including image content from said non-occluded image portion of said first image and image content from said first occluded image portion of the additional image includes:
including non-occluded image content in the majority of the first frame; and
including occluded image content in a minority of the first frame.
11. The method of claim 10 , wherein generating the first frame includes:
performing a downsampling operation on non-occluded image content to make room in said first frame for occluded image content.
12. The method of claim 1 , wherein said first frame is in a second frame format, the method further comprising:
generating a frame in a first frame format, said frame in the first frame format including image content from said non-occluded image portion of said first frame and no image content corresponding to an occluded portion of the environment; and
storing said frame in the first frame format in the storage device.
13. The method of claim 12 , wherein the first frame and the frame in the first frame format correspond to a first program, the method further comprising:
receiving a first request for content corresponding to the first program from a first playback device which supports the display of occluded image content; and
sending said first frame in the second frame format to the first device in response to said first request for content.
14. The method of claim 13 , further comprising:
wherein said UV map maps image content in said first frame corresponding to said non-occluded image portion of said first frame to a first segment of said environmental model which corresponds to a portion of said environment which is visible from said first location; and
wherein said UV map further maps image content in said first frame corresponding to said first occluded image portion to a second segment of said environmental model which corresponds to a portion of said environment which is not visible from said first location.
15. The method of claim 14 , further comprising:
transmitting to the second playback device an environmental model which does not include segments corresponding to portions of the environment occluded from view from the first location; and
transmitting to a second playback device a UV map to be used for mapping portions of the frame in the first format to the environmental model which does not include segments corresponding to portions of the environment occluded from view from the first location.
16. A non-transitory computer readable medium for use in a system, said non-transitory computer readable medium including computer executable instructions which, when executed by a computer, control the system to:
receive a first image corresponding to a portion of an environment, said first image including a non-occluded image portion corresponding to a portion of the environment visible from a first location;
receive an additional image of the environment including at least a first occluded image portion corresponding to a portion of the environment occluded from view from said first location;
generate a first frame including image content from said non-occluded image portion of said first image and image content from said first occluded image portion of the additional image; and
store said first frame in a storage device or transmit said first frame to another device.
17. An image processing system, comprising:
a processor configured to control said image processing system to:
receive a first image corresponding to a portion of an environment, said first image including a non-occluded image portion corresponding to a portion of the environment visible from a first location;
receive an additional image of the environment including at least a first occluded image portion corresponding to a portion of the environment occluded from view from said first location;
generate a first frame including image content from said non-occluded image portion of said first image and image content from said first occluded image portion of the additional image; and
store said first frame in a storage device or transmit said first frame to another device.
18. The image processing system of claim 17 , wherein said processor is further configured to control said image processing system to generate a UV map to be used for mapping portions of said first frame to segments of an environmental model.
19. The image processing system of claim 17 , wherein said processor is further configured to control said image processing system to generate an environmental model including segments corresponding to non-occluded surfaces of said environment and segments corresponding to occluded surfaces of said environment;
wherein said UV map maps image content in said first frame corresponding to said non-occluded image portion of said first frame to a first segment of said environmental model which corresponds to a portion of said environment which is visible from said first location; and
wherein said UV map further maps image content in said first frame corresponding to said first occluded image portion to a second segment of said environmental model which corresponds to a portion of said environment which is not visible from said first location.
20. The image processing system of claim 17 , wherein said first frame is in a second frame format; and
wherein said processor is further configured to control said image processing system to:
generate a frame in a first frame format, said frame in the first frame format including image content from said non-occluded image portion of said first frame and no image content corresponding to an occluded portion of the environment; and
store said frame in the first frame format in the storage device.
21. The image processing system of claim 20 , wherein the first frame and the frame in the first frame format correspond to a first program;
wherein said processor is further configured to control said image processing system to:
receive a first request for content corresponding to the first program from a first playback device which supports the display of occluded image content; and
send said first frame in the second frame format to the first device in response to said first request for content.
22. The image processing system of claim 21 , wherein said processor is further configured to:
control said image processing system to determine from device capability information, prior to sending the first frame in the second frame format, that the first playback device supports the display of occluded image content;
receive a second request for content corresponding to the first program from a second playback device which does not support the display of occluded image content; and
send said frame in the first frame format to the second playback device in response to said second request for content.