Methods and apparatus rendering images using point clouds representing one or more objects
A background image is also generated, e.g., by filling portions of a captured image where a foreground object was extracted and communicated to the playback device, Foreground objects are identified and point cloud representations of the foreground objects are generated and communicated to a playback device so that they can be used in generating images including the background which is communicated separately. In the case of a point cloud representation a number of points in an environment, e.g., 3D space, are communicated to the playback device along with color information. Thus in some embodiments a foreground object is represented as a set of points with corresponding color information on a per point basis. Foreground object information is communicated and processed in some embodiments at a different rate, e.g., faster rate, then the background textures. The playback device renders images which are sent to the display by first rendering a background layer using the communicated background information, e.g., background texture(s), UV map and environmental geometry, e.g., mesh, to which the background textures are applied.
1. A method for communicating point cloud information to a playback device, the method comprising:
obtaining a geometry model for an environment, a background image configured to be applied to the geometry model as a texture, and a set of point cloud information defining a foreground object in the environment; and
communicating, to the playback device, at least one updated set of point cloud information without the geometry model and the background image.
2. The method of claim 1 , further comprising:
communicating, to the playback device, a UV map comprising information indicative of how to apply the background image to the geometry model.
3. The method of claim 1 , wherein the set of point cloud information and the at least one updated set of point cloud information comprise information for a same number of points, wherein location information for one or more points in the at least one updated set of point cloud information differs from location information for one or more corresponding points in the set of point cloud information.
4. The method of claim 1 , wherein the set of point cloud information comprises location information and color information for a plurality of individual points which define the foreground object.
5. The method of claim 1 , further comprising:
generating the background image from a plurality of images captured at different times.
6. The method of claim 5 , wherein generating the background image from the plurality of images captured at different times comprises:
identifying the foreground object in a first image captured at a first time; and
replacing the foreground object in the first image with image content from a second image captured at a second time to form the background image.
7. The method of claim 6 wherein a position of the foreground object in the second image captured at the second time is different from a position of the foreground object in the first image captured at the first time.
8. The method of claim 1 , further comprising:
generating an additional updated set of point cloud information defining the foreground object;
communicating, to the playback device and at a first update frequency, the additional updated set of point cloud information; and
communicating, to the playback device and at a second update frequency, an updated background image configured to be applied to the geometry model as a texture, wherein the second update frequency is less than the first update frequency.
9. A system, comprising:
a memory; and
a processor coupled to the memory and configured to execute program instructions stored in the memory to cause the system to:
obtain a geometry model for an environment, a background image configured to be applied to the geometry model as a texture, and a set of point cloud information defining a foreground object in the environment; and
communicate, to a playback device, at least one updated set of point cloud information without the geometry model and the background image.
10. The system of claim 9 , wherein the processor is further configured to execute program instructions to cause the system to:
communicate, to the playback device, a UV map comprising information indicative of how to apply the background image to the geometry model.
11. The system of claim 9 , wherein the set of point cloud information and the at least one updated set of point cloud information comprise information for a same number of points, wherein location information for one or more points in the at least one updated set of point cloud information differs from location information for one or more corresponding points in the set of point cloud information.
12. The system of claim 9 , wherein the set of point cloud information comprises location information and color information for a plurality of individual points which define the foreground object.
13. The system of claim 9 , wherein the processor is further configured to execute program instructions to cause the system to:
generate the background image from a plurality of images captured at different times.
14. The system of claim 10 , wherein the processor is further configured to execute program instructions to cause the system to:
generate an additional updated set of point cloud information defining the foreground object;
communicate, to the playback device and at a first update frequency, the additional updated set of point cloud information; and
communicate, to the playback device and at a second update frequency, an updated background image configured to be applied to the geometry model as a texture, wherein the second update frequency is less than the first update frequency.
15. A system, comprising:
a memory; and
a processor coupled to the memory and configured to execute program instructions stored in the memory to cause the system to:
obtain a geometry model for an environment, a background image configured to be applied to the geometry model as a texture, and a set of point cloud information defining a foreground object in the environment;
communicate, to a playback device, at least one updated set of point cloud information without the geometry model and the background image;
generate an additional updated set of point cloud information defining the foreground object; and
communicate, to the playback device and at a first update frequency, the additional updated set of point cloud information.
16. The system of claim 15 , wherein the processor is further configured to execute program instructions to cause the system to:
communicate, to the playback device and at a second update frequency, an updated background image configured to be applied to the geometry model as a texture, wherein the second update frequency is less than the first update frequency.
17. The system of claim 16 , wherein the processor is further configured to execute program instructions to cause the system to:
communicate, to the playback device, a UV map comprising information indicative of how to apply the background image to the geometry model.
18. The system of claim 16 , wherein the set of point cloud information and the at least one updated set of point cloud information comprise information for a same number of points, wherein location information for one or more points in the at least one updated set of point cloud information differs from location information for one or more corresponding points in the set of point cloud information.
19. The system of claim 16 , wherein the set of point cloud information comprises location information and color information for a plurality of individual points which define the foreground object.
20. The system of claim 16 , wherein the processor is further configured to execute program instructions to cause the system to:
generate the background image from a plurality of images captured at different times.