Real-time mobile device capture and generation of AR/VR content
Various embodiments describe systems and processes for generating AR/VR content. In one aspect, a method for generating a three-dimensional (3D) projection of an object is provided. A sequence of images along a camera translation may be obtained using a single lens camera. Each image contains at least a portion of overlapping subject matter, which includes the object. The object is semantically segmented from the sequence of images using a trained neural network to form a sequence of segmented object images, which are then refined using fine-grained segmentation. On-the-fly interpolation parameters are computed and stereoscopic pairs are generated for points along the camera translation from the refined sequence of segmented object images for displaying the object as a 3D projection in a virtual reality or augmented reality environment. Segmented image indices are then mapped to a rotation range for display in the virtual reality or augmented reality environment.
1. A method for generating a three-dimensional (3D) projection of an object in a virtual reality or augmented reality environment, the method including:
obtaining a sequence of images using a single lens camera, the sequence of images being captured along a camera translation, wherein each image in the sequence of images contains at least a portion of overlapping subject matter, the subject matter including the object;
semantically segmenting the object from the sequence of images using a trained neural network to form a sequence of segmented object images, wherein the trained neural network is configured to aggregate a plurality of feature maps from different layers of the trained neural network in order to allow usage of both finer scale and coarser scale details to produce probability maps corresponding to the sequence of segmented object images, wherein the trained neural network is trained to label every pixel in each image in the sequence of images with a particular category label;
refining the sequence of segmented object images using fine-grained segmentation, wherein refining the sequence of segmented object images includes passing each probability map onto a temporal dense conditional random field (CRF) smoothing system to produce a binary mask for every segmented object image, wherein the binary masks are temporally consistent and sharply aligned at boundaries to each other;
computing on-the-fly interpolation parameters;
generating stereoscopic pairs from the refined sequence of segmented object images for displaying the object as a 3D projection in the virtual reality or augmented reality environment, the stereoscopic pairs being generated for one or more points along the camera translation; and
mapping segmented image indices to a rotation range for display in the virtual reality or augmented reality environment.
2. The method of claim 1 , wherein each image in the sequence of images is captured at a predetermined distance along the camera translation.
3. The method of claim 1 , further comprising stabilizing the sequence of images using camera rotation and focal length values before semantically segmenting the object from the sequence of images.
4. The method of claim 1 , wherein on-the-fly interpolation parameters are used to generate interpolated images along any point in the camera translation in real-time.
5. The method of claim 1 , wherein a stereoscopic pair includes an interpolated virtual image comprising two selected frames.
6. The method of claim 5 , wherein the selected frames are modified by rotating an image of a selected frame such that the image corresponds to a view of the object angled directly toward the object.
7. The method of claim 1 , wherein the sequence of segmented object images are fused to generate a projection of the object, the projection depicting a 3D view of the object without polygon generation.
8. The method of claim 1 , wherein using the dense temporal conditional random field includes utilizing a graph of neighboring images for each image to be refined.
9. The method of claim 1 , wherein mapping segmented image indices includes mapping physical viewing locations to a frame index.
10. A system for generating a three-dimensional (3D) projection of an object in a virtual reality or augmented reality environment, the system comprising:
a single lens camera for obtaining a sequence of images, the sequence of images being captured along a camera translation, wherein each image in the sequence of images contains at least a portion of overlapping subject matter, the subject matter including the object;
a display module;
a processor, and
memory storing one or more programs configured for execution by the processor, the one or more programs comprising instructions for:
semantically segmenting the object from the sequence of images using a trained neural network to form a sequence of segmented object images, wherein the trained neural network is configured to aggregate a plurality of feature maps from different layers of the trained neural network in order to allow usage of both finer scale and coarser scale details to produce probability maps corresponding to the sequence of segmented obj ect images, wherein the trained neural network is trained to label every pixel in each image in the sequence of images with a particular category label;
refining the sequence of segmented object images using fine-grained segmentation, wherein refining the sequence of segmented object images includes passing each probability map onto a temporal dense conditional random field (CRF) smoothing system to produce a binary mask for every segmented object image, wherein the binary masks are temporally consistent and sharply aligned at boundaries to each other;
computing on-the-fly interpolation parameters;
generating stereoscopic pairs from the refined sequence of segmented object images for displaying the object as a 3D projection in the virtual reality or augmented reality environment, the stereoscopic pairs being generated for one or more points along the camera translation; and
mapping segmented image indices to a rotation range for display in the virtual reality or augmented reality environment.
11. The system of claim 10 , wherein each image in the sequence of images is captured at a predetermined distance along the camera translation.
12. The system of claim 10 , further comprising stabilizing the sequence of images using camera rotation and focal length values before semantically segmenting the object from the sequence of images.
13. The system of claim 10 , wherein on-the-fly interpolation parameters are used to generate interpolated images along any point in the camera translation in real-time.
14. The system of claim 10 , wherein a stereoscopic pair includes an interpolated virtual image comprising two selected frames.
15. The system of claim 14 , wherein the selected frames are modified by rotating an image of a selected frame such that the image corresponds to a view of the object angled directly toward the object.
16. The system of claim 10 , wherein the sequence of segmented object images are fused to generate a projection of the object, the projection depicting a 3D view of the object without polygon generation.
17. The system of claim 10 , wherein using the dense temporal conditional random field includes utilizing a graph of neighboring images for each image to be refined.
18. A non-transitory computer readable medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:
obtaining a sequence of images using a single lens camera, the sequence of images being captured along a camera translation, wherein each image in the sequence of images contains at least a portion of overlapping subject matter, the subject matter including the object;
semantically segmenting the object from the sequence of images using a trained neural network to form a sequence of segmented object images, wherein the trained neural network is configured to aggregate a plurality of feature maps from different layers of the trained neural network in order to allow usage of both finer scale and coarser scale details to produce probability maps corresponding to the sequence of segmented object images, wherein the trained neural network is trained to label every pixel in each image in the sequence of images with a particular category label;
refining the sequence of segmented object images using fine-grained segmentation, wherein refining the sequence of segmented object images includes passing each probability map onto a temporal dense conditional random field (CRF) smoothing system to produce a binary mask for every segmented object image, wherein the binary masks are temporally consistent and sharply aligned at boundaries to each other;
computing on-the-fly interpolation parameters;
generating stereoscopic pairs from the refined sequence of segmented object images for displaying the object as a 3D projection in the virtual reality or augmented reality environment, the stereoscopic pairs being generated for one or more points along the camera translation; and
mapping segmented image indices to a rotation range for display in the virtual reality or augmented reality environment.