IP Library Granted Patent US 11,172,148
Granted Patent B2
US 11,172,148 · App. 16/736,469 · Granted Nov 9, 2021

Methods, systems, and media for generating compressed images

Inventor: Ryan Overbeck (Mountain View, CA)
Assignee: GOOGLE LLC
H04N5/3415G06T15/06G06T15/08G06T15/205
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,172,148
App. No.
16/736,469
Granted
Nov 9, 2021
Kind
B2
Abstract

Methods, systems, and media for generating compressed images are provided. In some embodiments, the method comprises: identifying a multi-plane image (MPI) that represents a three-dimensional image; splitting the MPI into a plurality of sub-volumes; calculating, for each sub-volume of the MPI, a depthmap; converting each depthmap to a mesh, wherein each mesh corresponds to a layer of a plurality of layers associated with a multi-depth image (MDI) to be rendered; calculating, for each layer of the plurality of layers, an image that indicates a color and a transmittance of each voxel included in the layer; storing the meshes corresponding to the plurality of layers of the MDI and the images corresponding to the plurality of layers of the MDI as the MDI; and, in response to receiving a request for the three-dimensional image from a user device, transmitting the MDI to the user device, wherein the user device is configured to render the MDI by mapping, for each layer of the MDI, the image corresponding to the layer as a texture on the mesh corresponding to the layer.

Claims (43)

1. A method for generating compressed images, the method comprising:

identifying a multi-plane image (MPI) that represents a three-dimensional image, wherein the MPI comprises a plurality of fronto-parallel planes, each associated with an image that indicates a color and a transmittance of each pixel of the plurality of fronto-parallel planes;

splitting the MPI into a plurality of sub-volumes, wherein each sub-volume in the plurality of sub-volumes includes a subset of the plurality of fronto-parallel planes;

calculating, for each sub-volume of the MPI, a depthmap;

converting each depthmap to a mesh, wherein each mesh corresponds to a layer of a plurality of layers associated with a multi-depth image (MDI) to be rendered, wherein each depthmap indicates a location and a depth of each voxel of the MDI included in the corresponding layer of the plurality of layers associated with the MDI, and wherein a number of layers in the plurality of layers associated with the MDI is less than a number of fronto-parallel planes included in the plurality of fronto-parallel planes associated with the MPI;

calculating, for each layer of the plurality of layers, an image that indicates a color and a transmittance of each voxel included in the layer;

storing the meshes corresponding to the plurality of layers of the MDI and the calculated images corresponding to the plurality of layers of the MDI as the MDI; and

in response to receiving a request for the three-dimensional image from a user device, transmitting the MDI to the user device, wherein the user device is configured to render the MDI by mapping, for each layer of the MDI, the calculated image corresponding to the layer as a texture on the mesh corresponding to the layer.

2. The method of claim 1 , further comprising generating a sequence of MDI images corresponding to a sequence of MPI images, wherein the sequence of MPI images corresponds to three-dimensional video content.

3. The method of claim 1 , wherein splitting the MPI into the plurality of sub-volumes comprises optimizing a plurality of cuts that generate the plurality of sub-volumes by minimizing a rendering error generated by rendering the MDI using the plurality of sub-volumes.

4. The method of claim 3 , wherein the rendering error comprises a unary term that indicates an error in depth resulting from rendering the MDI using a cut of the plurality of cuts.

5. The method of claim 3 , wherein the rendering error comprises a smoothness term that indicates a smoothness of a cut of the plurality of cuts across voxels included in the sub-volume corresponding to the cut.

6. The method of claim 1 , wherein splitting the MPI into the plurality of sub-volumes comprises using a trained neural network to identify a plurality of cuts that generate the plurality of sub-volumes.

7. The method of claim 1 , wherein each mesh corresponding to each layer of the MDI is a triangular mesh.

8. A system for generating compressed images, the system comprising:

a hardware processor that is configured to:

identify a multi-plane image (MPI) that represents a three-dimensional image, wherein the MPI comprises a plurality of fronto-parallel planes, each associated with an image that indicates a color and a transmittance of each pixel of the plurality of fronto-parallel planes;

split the MPI into a plurality of sub-volumes, wherein each sub-volume in the plurality of sub-volumes includes a subset of the plurality of fronto-parallel planes;

calculate, for each sub-volume of the MPI, a depthmap;

convert each depthmap to a mesh, wherein each mesh corresponds to a layer of a plurality of layers associated with a multi-depth image (MDI) to be rendered, wherein each depthmap indicates a location and a depth of each voxel of the MDI included in the corresponding layer of the plurality of layers associated with the MDI, and wherein a number of layers in the plurality of layers associated with the MDI is less than a number of fronto-parallel planes included in the plurality of fronto-parallel planes associated with the MPI;

calculate, for each layer of the plurality of layers, an image that indicates a color and a transmittance of each voxel included in the layer;

store the meshes corresponding to the plurality of layers of the MDI and the calculated images corresponding to the plurality of layers of the MDI as the MDI; and

in response to receiving a request for the three-dimensional image from a user device, transmit the MDI to the user device, wherein the user device is configured to render the MDI by mapping, for each layer of the MDI, the calculated image corresponding to the layer as a texture on the mesh corresponding to the layer.

9. The system of claim 8 , wherein the hardware processor is further configured to generate a sequence of MDI images corresponding to a sequence of MPI images, wherein the sequence of MPI images corresponds to three-dimensional video content.

10. The system of claim 8 , wherein splitting the MPI into the plurality of sub-volumes comprises optimizing a plurality of cuts that generate the plurality of sub-volumes by minimizing a rendering error generated by rendering the MDI using the plurality of sub-volumes.

11. The system of claim 10 , wherein the rendering error comprises a unary term that indicates an error in depth resulting from rendering the MDI using a cut of the plurality of the cuts.

12. The system of claim 10 , wherein the rendering error comprises a smoothness term that indicates a smoothness of a cut of the plurality of cuts across voxels included in the sub-volume corresponding to the cut.

13. The system of claim 8 , wherein splitting the MPI into the plurality of sub-volumes comprises using a trained neural network to identify a plurality of cuts that generate the plurality of sub-volumes.

14. The system of claim 8 , wherein each mesh corresponding to each layer of the MDI is a triangular mesh.

15. A non-transitory computer-readable medium containing computer executable instructions that, when executed by a processor, cause the processor to perform a method for generating compressed images, the method comprising:

identifying a multi-plane image (MPI) that represents a three-dimensional image, wherein the MPI comprises a plurality of fronto-parallel planes, each associated with an image that indicates a color and a transmittance of each pixel of the plurality of fronto-parallel planes;

splitting the MPI into a plurality of sub-volumes, wherein each sub-volume in the plurality of sub-volumes includes a subset of the plurality of fronto-parallel planes;

calculating, for each sub-volume of the MPI, a depthmap;

converting each depthmap to a mesh, wherein each mesh corresponds to a layer of a plurality of layers associated with a multi-depth image (MDI) to be rendered, wherein each depthmap indicates a location and a depth of each voxel of the MDI included in the corresponding layer of the plurality of layers associated with the MDI, and wherein a number of layers in the plurality of layers associated with the MDI is less than a number of fronto-parallel planes included in the plurality of fronto-parallel planes associated with the MPI;

calculating, for each layer of the plurality of layers, an image that indicates a color and a transmittance of each voxel included in the layer;

storing the meshes corresponding to the plurality of layers of the MDI and the calculated images corresponding to the plurality of layers of the MDI as the MDI; and

in response to receiving a request for the three-dimensional image from a user device, transmitting the MDI to the user device, wherein the user device is configured to render the MDI by mapping, for each layer of the MDI, the calculated image corresponding to the layer as a texture on the mesh corresponding to the layer.

16. The non-transitory computer-readable medium of claim 15 , wherein the method further comprises generating a sequence of MDI images corresponding to a sequence of MPI images, wherein the sequence of MPI images corresponds to three-dimensional video content.

17. The non-transitory computer-readable medium of claim 15 , wherein splitting the MPI into the plurality of sub-volumes comprises optimizing a plurality of cuts that generate the plurality of sub-volumes by minimizing a rendering error generated by rendering the MDI using the plurality of sub-volumes.

18. The non-transitory computer-readable medium of claim 17 , wherein the rendering error comprises a unary term that indicates an error in depth resulting from rendering the MDI using a cut of the plurality of cuts.

19. The non-transitory computer-readable medium of claim 17 , wherein the rendering error comprises a smoothness term that indicates a smoothness of a cut of the plurality of cuts across voxels included in the sub-volume corresponding to the cut.

20. The non-transitory computer-readable medium of claim 15 , wherein splitting the MPI into the plurality of sub-volumes comprises using a trained neural network to identify a plurality of cuts that generate the plurality of sub-volumes.

21. The non-transitory computer-readable medium of claim 15 , wherein each mesh corresponding to each layer of the MDI is a triangular mesh.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 9, 2020
From: OVERBECK, RYAN
To: GOOGLE LLC
Reel/Frame 051460/0940 →
Continuity (1)
Related Publication 20210211593A1 · Jul 8, 2021
Cited By (2)
US 12,470,681 US 12,561,777