Methods and apparatus rendering images using point clouds representing one or more objects
A background image is also generated, e.g., by filling portions of a captured image where a foreground object was extracted and communicated to the playback device, Foreground objects are identified and point cloud representations of the foreground objects are generated and communicated to a playback device so that they can be used in generating images including the background which is communicated separately. In the case of a point cloud representation a number of points in an environment, e.g., 3D space, are communicated to the playback device along with color information. Thus in some embodiments a foreground object is represented as a set of points with corresponding color information on a per point basis. Foreground object information is communicated and processed in some embodiments at a different rate, e.g., faster rate, then the background textures. The playback device renders images which are sent to the display by first rendering a background layer using the communicated background information, e.g., background texture(s), UV map and environmental geometry, e.g., mesh, to which the background textures are applied.
1. A method comprising:
obtaining, for an environment comprising a foreground object in front of a background, background geometry information of the environment, and a representation of the foreground object in the environment; and
communicating, to a playback device, an updated representation of the foreground object without the background geometry information, wherein the updated representation of the foreground object is configured to be rendered as a foreground layer over a background layer comprising the background geometry information.
2. The method of claim 1 , further comprising:
obtaining a background image configured to be applied to the background geometry information as a texture,
wherein the updated representation of the foreground object is communicated without the background image.
3. The method of claim 2 , further comprising:
communicating, to the playback device, a UV map comprising information indicative of how to apply the background image to the background geometry information.
4. The method of claim 2 , further comprising:
generating the background image from a plurality of images captured at different times.
5. The method of claim 4 , wherein generating the background image comprises:
identifying the foreground object in a first image captured at a first time; and
replacing the foreground object in the first image with image content from a second image captured at a second time to form the background image.
6. The method of claim 1 , wherein the representation of the foreground object and the updated representation of the foreground object is indicative of change in location of the foreground object.
7. The method of claim 1 , wherein the representation of the foreground object comprises image data and geometry data for the foreground object.
8. A non-transitory computer readable medium comprising computer readable code executable by one or more processors to:
obtain, for an environment comprising a foreground object in front of a background, background geometry information of the environment, and a representation of the foreground object in the environment; and
communicate, to a playback device, an updated representation of the foreground object without the background geometry information, wherein the updated representation of the foreground object is configured to be rendered as a foreground layer over a background layer comprising the background geometry information.
9. The non-transitory computer readable medium of claim 8 , further comprising computer readable code to:
obtain a background image configured to be applied to the background geometry information as a texture,
wherein the updated representation of the foreground object is communicated without the background image.
10. The non-transitory computer readable medium of claim 9 , further comprising computer readable code to:
communicate, to the playback device, a UV map comprising information indicative of how to apply the background image to the background geometry information.
11. The non-transitory computer readable medium of claim 9 , further comprising computer readable code to:
generate the background image from a plurality of images captured at different times.
12. The non-transitory computer readable code of claim 11 , wherein the computer readable code to generate the background image comprises computer readable code to:
identify the foreground object in a first image captured at a first time; and
replace the foreground object in the first image with image content from a second image captured at a second time to form the background image.
13. The non-transitory computer readable code of claim 8 , wherein the representation of the foreground object and the updated representation of the foreground object is indicative of change in location of the foreground object.
14. The non-transitory computer readable code of claim 8 , wherein the representation of the foreground object comprises image data and geometry data for the foreground object.
15. A system comprising:
one or more processors; and
one or more computer readable media comprising computer readable code executable by the one or more processors to:
obtain, for an environment comprising a foreground object in front of a background, background geometry information of the environment, and a representation of the foreground object in the environment; and
communicate, to a playback device, an updated representation of the foreground object without the background geometry information, wherein the updated representation of the foreground object is configured to be rendered as a foreground layer over a background layer comprising the background geometry information.
16. The system of claim 15 , further comprising computer readable code to:
obtain a background image configured to be applied to the background geometry information as a texture,
wherein the updated representation of the foreground object is communicated without the background image.
17. The system of claim 16 , further comprising computer readable code to:
communicate, to the playback device, a UV map comprising information indicative of how to apply the background image to the background geometry information.
18. The system of claim 16 , further comprising computer readable code to:
generate the background image from a plurality of images captured at different times.
19. The system of claim 15 , wherein the representation of the foreground object and the updated representation of the foreground object is indicative of change in location of the foreground object.
20. The system of claim 15 , wherein the representation of the foreground object comprises image data and geometry data for the foreground object.