Capture of raw augmented reality data and subsequent 3D mesh generation
Embodiments include systems and methods for generating a 3D mesh from a video stream or other image captured contemporaneously with AR data. The AR data is used to create a depth map, which is then fused with images from frames of the video to form a full 3D mesh. The images and depth map can also be used with an object detection algorithm to recognize 3D objects within the 3D mesh. Methods for fingerprinting the video with AR data captured contemporaneously with each frame are disclosed.
1. A method for generation of an abstract video from a video stream, comprising:
capturing a video stream and AR data synchronized to the video stream, wherein the video stream is comprised a plurality of frames, each frame including image data;
generating, with the video stream and AR data, a depth map for each frame of the plurality of frames of the video stream;
incrementally generating, from the depth map and image data for each frame of the plurality of frames, a 3D volume representing an environment captured by the video stream;
detecting at least one object from the plurality of frames; and
generating, from the 3D volume and the at least one detected object, an abstracted video.
2. The method of claim 1 , wherein capturing the video stream and AR data synchronized to the video stream further comprises:
generating, for each frame of the plurality of frames, a fingerprint that is unique to each frame;
associating each fingerprint with its corresponding frame; and
associating each fingerprint with AR data, from the captured AR data, that corresponds to the corresponding frame.
3. The method of claim 1 , further comprising:
correlating each of the detected objects with the abstract video to obtain a predicted 3D position and 3D rotation of each object; and
generating, with the detected objects, a 3D scene graph.
4. The method of claim 3 , further comprising obtaining the predicted 3D position and 3D rotation of each object using a neural network.
5. The method of claim 3 , further comprising referencing the one or more objects with a database of 3D models.
6. The method of claim 1 , wherein capturing the video stream and AR data comprises capturing the video stream with a video camera, and capturing at least some of the AR data with one or more spatial position sensors.
7. The method of claim 1 , further comprising calculating at least some of the AR data from the plurality of frames.
8. A non-transitory computer readable medium (CRM) comprising instructions that, when executed by an apparatus, cause the apparatus to:
capture a video stream comprised of a plurality of frames, each frame including image data;
capture AR data corresponding to each frame of the plurality of frames;
generate, from the AR data, a depth map for each frame of the plurality of frames;
construct, from the depth map of each frame, a 3D volume;
detect, within the video stream, at least one object; and
generate, from the 3D volume and the at least one detected object, an abstracted video.
9. The CRM of claim 8 , wherein the instructions are to further cause the apparatus to:
generate for each frame of the plurality of frames, a fingerprint that is unique to each frame;
tag each fingerprint to its frame; and
associate each frame's corresponding AR data to its fingerprint.
10. The CRM of claim 8 , wherein the instructions are to further cause the apparatus to:
correlate each of the detected objects with the abstracted video to obtain a predicted 3D position and 3D rotation of each object; and
generate, with the detected objects, a 3D scene graph.
11. The CRM of claim 8 , wherein the instructions are to further cause the apparatus to reference the one or more detected objects with a database of 3D models.
12. The CRM of claim 8 , wherein the instructions are to further cause the apparatus to employ a neural net to generate the depth map for each frame of the plurality of frames.
13. The CRM of claim 8 , wherein the instructions are to further cause the apparatus to capture the video stream from a video camera, and capture the AR data at least in part with one or more spatial position sensors.
14. The CRM of claim 13 , wherein the instructions are to further cause the apparatus to capture the AR data corresponding to each frame of the plurality of frames at least in part from analysis of each corresponding frame.
15. The CRM of claim 8 , wherein the apparatus is a mobile device.