Methods and apparatus to synthesize six degree-of-freedom views from sparse RGB-depth inputs
Methods and apparatus are disclosed herein to synthesize six degree-of-freedom views from sparce RGB-depth inputs. An example apparatus includes at least one memory, instructions in the apparatus, and processor circuitry to execute the instructions to reproject a first red-green-blue (RGB) depth view and a second RGB depth view to a target camera position to obtain a first reprojected view and a second reprojected view, combine the first reprojected view and the second reprojected view into a blended view data based on a first weight map and a second weight map, the blended view including missing RGB depth information due to at least one of an occlusion or a disocclusion, and generate a six degree-of-freedom synthesized view of the blended view data, the synthesized view including the missing RGB depth information.
1 . An apparatus comprising:
at least one memory;
instructions in the apparatus; and
at least one programmable circuit to be programmed based on the instructions to:
train a convolutional neural network and a generative adversarial network jointly, the convolutional neural network to generate weight maps, the generative adversarial network to generate synthesized views;
reproject a first red-green-blue (RGB) depth (RGBD) view and a second RGBD view to a target camera position to obtain a first reprojected view and a second reprojected view;
combine the first reprojected view and the second reprojected view into a blended view based on a first weight map and a second weight map, the first weight map and the second weight map based on the convolutional neural network, the blended view associated with missing information due to at least one of an occlusion or a disocclusion; and
cause the generative adversarial network to generate a first synthesized view based on the blended view, the first synthesized view to include synthesized information to fill in the missing information.
2 . The apparatus of claim 1 , wherein one or more of the at least one programmable circuit is to reproject the first RGBD view and the second RGBD view based on camera parameters, the camera parameters including at least one of an extrinsic camera parameter or an intrinsic camera parameter.
3 . The apparatus of claim 1 , wherein one or more of the at least one programmable circuit is to train the convolutional neural network using an L1 loss function.
4 . The apparatus of claim 1 , wherein one or more of the at least one programmable circuit is to train the generative adversarial network using at least one of an L1 loss function, a VGG loss function, or a GAN loss function.
5 . The apparatus of claim 1 , wherein one or more of the at least one programmable circuit is to generate the first synthesized view based on a virtual viewpoint moving trajectory with a sequence of target positions or rotation matrices.
6 . A method comprising:
training a convolutional neural network and a generative adversarial network jointly, the convolutional neural network to generate weight maps, the generative adversarial network to generate synthesized views;
reprojecting a first red-green-blue (RGB) depth (RGBD) view and a second RGBD view to a target camera position to obtain a first reprojected view and a second reprojected view;
combining the first reprojected view and the second reprojected view into a blended view based on a first weight map and a second weight map, the first weight map and the second weight map based on the convolutional neural network, the blended view associated with missing information due to at least one of an occlusion or a disocclusion; and
generating, with the generative adversarial network, a first synthesized view based on the blended view, the first synthesized view including synthesized information to fill in the missing information.
7 . The method of claim 6 , including reprojecting the first RGBD view and the second RGBD view based on camera parameters, the camera parameters including at least one of an extrinsic camera parameter or an intrinsic camera parameter.
8 . The method of claim 6 , including training the convolutional neural network using an L1 loss function.
9 . The method of claim 6 , including training the generative adversarial network using at least one of an L1 loss function, a VGG loss function, or a GAN loss function.
10 . The method of claim 6 , including generating the first synthesized view based on a virtual viewpoint moving trajectory with a sequence of target positions or rotation matrices.
11 . A non-transitory computer readable storage medium comprising computer readable instructions to cause at least one programmable circuit to at least:
train a convolutional neural network and a generative adversarial network jointly, the convolutional neural network to generate weight maps, the generative adversarial network to generate synthesized views;
reproject a first red-green-blue (RGB) depth (RGBD) view and a second RGBD view to a target camera position to obtain a first reprojected view and a second reprojected view;
combine the first reprojected view and the second reprojected view into a blended view based on a first weight map and a second weight map, the first weight map and the second weight map based on the convolutional neural network, the blended view associated with missing information due to at least one of an occlusion or a disocclusion; and
cause the generative adversarial network to generate a first synthesized view based on the blended view, the first synthesized view to include synthesized information to fill in the missing information.
12 . The non-transitory computer readable storage medium as defined in claim 11 , wherein the computer readable instructions are to cause one or more of the at least one programmable circuit to reproject the first RGBD view and the second RGBD view based on camera parameters, the camera parameters including at least one of an extrinsic camera parameter or an intrinsic camera parameter.
13 . The non-transitory computer readable storage medium as defined in claim 11 , wherein the computer readable instructions are to cause one or more of the at least one programmable circuit to train the convolutional neural network using an L1 loss function.
14 . The non-transitory computer readable storage medium as defined in claim 11 , wherein the computer readable instructions are to cause one or more of the at least one programmable circuit to train the generative adversarial network using at least one of an L1 loss function, a VGG loss function, or a GAN loss function.
15 . The non-transitory computer readable storage medium as defined in claim 11 , wherein the computer readable instructions are to cause one or more of the at least one programmable circuit to generate the first synthesized view based on a virtual viewpoint moving trajectory with a sequence of target positions or rotation matrices.