Method and apparatus for supporting augmented and/or virtual reality playback using tracked objects
Methods for capturing and generating information about objects in a 3D environment that can be used to support augmented reality or virtual reality playback operations in a data efficient manner are described. In various embodiments one or more frames including foreground objects are generated and transmitted with corresponding information that can be used to determine the location where the foreground objects are to be positioned relative to a background for one or more frame times are described. Data efficiency is achieved by specifying different locations for a foreground object for different frame times avoiding in some embodiments the need to transmit an image and depth information defining the same of the foreground for each frame time. The frames can be encoded using a video encoder even though some of the information communicated are not pixel values but alpha blending values, object position information, mesh distortion information, etc.
1. A method of operating a content delivery system, the method comprising:
extracting a foreground object image showing a foreground object from a captured image;
generating mesh distortion information for the foreground object, wherein the mesh distortion information comprises data indicative of distortion to be applied to a default object mesh to obtain a distorted object mesh, wherein the distorted object mesh represents a shape of the foreground object;
packing the foreground object image and the mesh distortion information into a frame; and
transmitting the frame to a playback device.
2. The method of claim 1 , further comprising:
packing a set of alpha blending coefficients into the frame.
3. The method of claim 1 , wherein the captured image comprises a plurality of foreground objects, wherein for each unique foreground object in the plurality of foreground objects the method further comprises:
extracting a unique foreground object image showing the unique foreground object from the captured image;
generating a unique set of mesh distortion information for the unique foreground object, wherein the unique set of mesh distortion information comprises data indicative of a distortion to be applied to the default object mesh to obtain a distorted object mesh, wherein the distorted object mesh represents a shape of the unique foreground object; and
packing the unique foreground object image and the unique set of mesh distortion information into the frame.
4. The method of claim 1 , further comprising:
generating object placement information indicative of a display position for the foreground object image; and
packing the object placement information into the frame.
5. A content delivery system, the system comprising:
a non-transitory computer readable medium;
at least one processor coupled to the non-transitory computer readable medium and configured to execute instructions stored in the non-transitory computer readable medium which cause the at least one processor to:
extract a foreground object image showing a foreground object from a captured image;
generate mesh distortion information for the foreground object, wherein the mesh distortion information comprises data indicative of a distortion to be applied to a default object mesh to obtain a distorted object mesh, wherein the distorted object mesh represents a shape of the foreground object;
pack the foreground object image and the mesh distortion information into a frame; and
transmit the frame to a playback device.
6. A playback system, comprising:
at least one non-transitory computer-readable medium comprising computer instructions; and
a processor coupled to the at least non-transitory computer-readable medium and configured to execute the computer instructions which cause the processor to:
obtain a frame comprising textures of a plurality of foreground objects and corresponding object mesh distortion information, wherein the object mesh distortion information comprises data indicative of a distortion to be applied to a default object mesh to obtain distorted object meshes, wherein the distorted object meshes represent shapes of the foreground objects;
for each unique foreground object in the plurality of foreground objects:
distort the default object mesh based on the corresponding object mesh distortion information;
apply the texture of the foreground object to the distorted object mesh to obtain a foreground object image; and
generate an output image based on the plurality of foreground object image.
7. The content delivery system of claim 5 , wherein the at least one processor is further configured to execute instructions which cause the at least one processor to pack a set of alpha blending coefficients into the frame.
8. The content delivery system of claim 5 , wherein the captured image comprises a plurality of foreground objects, wherein the at least one processor is further configured to execute instructions which cause the at least one processor to, for each unique foreground object in the plurality of foreground objects:
extract a unique foreground object image showing the unique foreground object from the captured image;
generate a unique set of mesh distortion information for the unique foreground object, wherein the unique set of mesh distortion information comprises data indicative of a distortion to be applied to the default object mesh to obtain a distorted object mesh, wherein the distorted object mesh represents a shape of the unique foreground object;
pack the unique foreground object image and the unique set of mesh distortion information into the frame.
9. The content delivery system of claim 5 , wherein the at least one processor is further configured to execute instructions which cause the at least one processor to:
generate object placement information indicative of a display position for the foreground object image; and
pack the object placement information into the frame.
10. The playback system of claim 6 , wherein the processor is further configured to execute computer instructions which cause the processor to obtain, for each unique foreground object in the plurality of foreground objects, object placement information indicative of a display position for the unique foreground object, and wherein the processor is further configured to generate the output image based on the object placement information.