Viewpoint synthesis with enhanced 3D perception
Systems and methods relate to segmenting texture data and depth data of first image data of a first viewpoint into layers; generating respective multiple focal planes (MFPs) for each respective layer; blanking out pixels on the respective MFPs for each respective layer that are occluded by pixels on layers that are closer to a first viewpoint; and generating second image data for a second viewpoint to enable display of the second image data by: shifting and scaling the respective MFPs for each respective layer corresponding to the second viewpoint, wherein layers closer to an origin of the second viewpoint are shifted and scaled more than layers farther from the origin of the second viewpoint; and blanking out pixels on the shifted and scaled respective MFPs for each respective layer that are occluded by pixels on layers that are closer to the second viewpoint.
1. A method comprising:
accessing first image data comprising texture data and depth data from a first viewpoint;
segmenting the texture data and the depth data into a plurality of layers;
generating respective multiple focal planes (MFPs) for each respective layer based on the segmented texture data and the segmented depth data;
blanking out pixels on the respective MFPs for each respective layer that are occluded by pixels on layers that are closer to the first viewpoint;
identifying a second viewpoint; and
generating second image data for the second viewpoint to enable display of the second image data by:
shifting and scaling the respective MFPs for each respective layer corresponding to the second viewpoint, wherein layers closer to an origin of the second viewpoint are shifted and scaled more than layers farther from the origin of the second viewpoint; and
blanking out pixels on the shifted and scaled respective MFPs for each respective layer that are occluded by pixels on layers that are closer to the second viewpoint.
2. The method of claim 1 , wherein accessing the first image data comprises:
receiving coded data and decoding the coded data to the texture data and the depth data.
3. The method of claim 1 , wherein the texture data and the depth data from the first viewpoint comprises:
first texture data and first depth data from the first viewpoint captured at a first time, and
second texture data and second depth data from the first viewpoint captured at a second time.
4. The method of claim 1 , wherein the first image data further comprises one or more additional texture data and additional depth data from one or more additional viewpoints.
5. The method of claim 1 , wherein the generating the second image data further comprises:
filling in at least one hole in at least one of the shifted and scaled respective MFPs using data from a same depth layer.
6. The method of claim 1 , wherein the generating the second image data further comprises:
segmenting one or more objects of the first image data from the first viewpoint on each layer; and
filling in at least one hole generated from moving an occluding object in at least one of the shifted and scaled respective MFPs using a copy of the occluding object.
7. The method of claim 6 , wherein the filling in the at least one hole generated from moving the occluding object comprises:
setting a luminance value of a missing pixel to a luminance value calculated by blending functions using a new depth of a corresponding pixel of a moved copy of the occluding object to fill the at least one hole; and
setting a color value of the missing pixel to a color value of a corresponding pixel of the occluding object.
8. The method of claim 6 , wherein the filling in the at least one hole generated from moving the occluding object comprises:
setting a luminance value of a missing pixel to a luminance value calculated by interpolating and/or extrapolating pixel luminance values on hole edges; and
setting a color value of the missing pixel to a color value calculated by interpolating and/or extrapolating pixel color values on the hole edges.
9. The method of claim 1 , wherein the blanking out pixels on the respective MFPs and the blanking out pixels on the shifted and scaled respective MFPs comprises setting pixel luminance values to zero.
10. The method of claim 1 , further comprising:
tracking a position of a user, wherein the identifying the second viewpoint is based on the tracking.
11. A system comprising:
input/output circuitry configured to:
access first image data comprising texture data and depth data from a first viewpoint; and
control circuitry configured to:
segment the texture data and the depth data into a plurality of layers;
generate respective multiple focal planes (MFPs) for each respective layer based on the segmented texture data and the segmented depth data;
blank out pixels on the respective MFPs for each respective layer that are occluded by pixels on layers that are closer to the first viewpoint;
identify a second viewpoint; and
generate second image data for the second viewpoint to enable display of the second image data by:
shifting and scaling the respective MFPs for each respective layer corresponding to the second viewpoint, wherein layers closer to an origin of the second viewpoint are shifted and scaled more than layers farther from the origin of the second viewpoint; and
blanking out pixels on the shifted and scaled respective MFPs for each respective layer that are occluded by pixels on layers that are closer to the second viewpoint.
12. The system of claim 11 , wherein:
the input/output circuitry is configured to access the first image data by receiving coded data; and
the control circuitry is further configured to decode the coded data to the texture data and the depth data.
13. The system of claim 11 , wherein the texture data and the depth data from the first viewpoint comprises:
first texture data and first depth data from the first viewpoint captured at a first time, and
second texture data and second depth data from the first viewpoint captured at a second time.
14. The system of claim 11 , wherein the first image data further comprises one or more additional texture data and additional depth data from one or more additional viewpoints.
15. The system of claim 11 , wherein the control circuitry is further configured to generate the second image data by:
filling in at least one hole in at least one of the shifted and scaled respective MFPs using data from a same depth layer.
16. The system of claim 11 , wherein the control circuitry is further configured to generate the second image data by:
segmenting one or more objects of the first image data from the first viewpoint on each layer; and
filling in at least one hole generated from moving an occluding object in at least one of the shifted and scaled respective MFPs using a copy of the occluding object.
17. The system of claim 16 , wherein the filling in the at least one hole generated from moving the occluding object comprises:
setting a luminance value of a missing pixel to a luminance value calculated by blending functions using a new depth of a corresponding pixel of a moved copy of the occluding object to fill the at least one hole; and
setting a color value of the missing pixel to a color value of a corresponding pixel of the occluding object.
18. The system of claim 16 , wherein the filling in the at least one hole generated from moving the occluding object comprises:
setting a luminance value of a missing pixel to a luminance value calculated by interpolating and/or extrapolating pixel luminance values on hole edges; and
setting a color value of the missing pixel to a color value calculated by interpolating and/or extrapolating pixel color values on the hole edges.
19. The system of claim 11 , wherein the control circuitry is configured to blank out pixels on the respective MFPs and to blank out pixels on the shifted and scaled respective MFPs by setting pixel luminance values to zero.
20. The system of claim 11 , wherein the control circuitry is further configured to:
track a position of a user, wherein the control circuitry is configured to identify the second viewpoint based on the tracking.