Generating 3D representations of physical environments using key frames
Various implementations disclosed herein include devices, systems, and methods that use frames of sensor data to create key frames that are used to provide a 3D representation of a physical environment (e.g., a 3D mesh, planar representations, object bounding boxes, a combo of these, etc.). Some implementations use key frames (e.g., bundles of 2D image data deemed to be representative or useful for reconstruction/3D representation generation). Such key frames may be created by evaluating multiple frames of sensor data and determining whether to convert a given frame into a key frame or fuse it, e.g., fuse it with a nearest existing key frame. Such fusing may reduce loss of information which would occur if frames not selected as key frames are simply discarded.
1 . A method comprising:
at a device having a processor:
obtaining a plurality of frames of sensor data of a physical environment, the frames of sensor data captured while the device is moved along a path of multiple positions and multiple orientations within the physical environment;
identifying a first frame of the plurality of frames as a key frame based on a new key frame selection criterion, wherein the new key frame selection criterion assesses similarity between an evaluated frame and one or more previously-identified key frames;
determining to fuse a second frame of the plurality of frames with the key frame based on the second frame not being identified as another key frame based on the new key frame selection criterion;
fusing the second frame of the plurality of frames with the key frame, wherein the key frame aggregates data from the first frame and the second frame; and
generating a 3D representation of the physical environment based on the key frame.
2 . The method of claim 1 , wherein the sensor data comprises depth images, the depth images comprising pixel values corresponding to distances of portions of the physical environment from sensor viewpoint positions along the path.
3 . The method of claim 1 , wherein the sensor data comprises light intensity images, the light intensity images comprising pixel values corresponding to light intensity of light reflected from portions of the physical environment from sensor viewpoint positions along the path.
4 . The method of claim 1 , wherein the sensor data comprises semantic segmentation images, the semantic segmentation images comprising pixel values corresponding to object types of portions of the physical environment from sensor viewpoint positions along the path.
5 . The method of claim 1 , wherein the new key frame selection criterion assesses proximity between poses associated with an evaluated frame and one or more previously-identified key frames.
6 . The method of claim 1 , wherein the fusing is performed by a spatiotemporal fusion engine configured to fuse data from a sequence of images.
7 . The method of claim 1 , wherein the 3D representation is generated in real-time as the sensor data is captured while the device is moved along the path within the physical environment.
8 . The method of claim 1 , wherein the 3D representation is a parametric representation of 3D positions of one or more planes of the physical environment.
9 . The method of claim 1 , wherein the 3D representation is a bounding box representation of 3D positions of one or more objects of the physical environment.
10 . The method of claim 1 , wherein generating the 3D representation comprises:
generating a voxel-based representation of the physical environment based on a set of key frames, the set of key frames including the key frame; and
generating a mesh-based representation of the physical environment based on the voxel-based representation.
11 . The method of claim 1 , wherein the 3D representation is generated based an adaptive process in which key frame fusion is only performed during periods in which device movement is above a threshold.
12 . The method of claim 1 , wherein the 3D representation is generated based an adaptive process in which:
key frames generated from the plurality of frames are used during periods during which device movement is above a threshold; and
raw frames of the plurality of frames are used during periods during which the device movement is below the threshold.
13 . The method of claim 1 , wherein the plurality of frames are produced by a simultaneous localization and mapping (SLAM) process, wherein the SLAM process tracks the motion of the device using odometry between SLAM frame captures and the SLAM process provides frame-to-frame motion data for use in generating the 3D reconstruction.
14 . The method of claim 13 , wherein the SLAM process provides correction data that corrects previously-provided frame-to-frame motion data, wherein the 3D representation is adjusted based on the correction data.
15 . The method of claim 1 , wherein the 3D representation is generated based on generating a voxel-based representation, wherein voxels within the voxel-based representation are updated selectively based on an assessment of relevance of a key frame to respective voxels.
16 . The method of claim 15 , wherein a simultaneous localization and mapping (SLAM) process provides a drift correction, wherein only a subset of less than all of the voxels are updated based on the drift correction.
17 . The method of claim 1 , wherein the 3D representation is updated based a single anchor with respect to which all key frames poses are transformed.
18 . The method of claim 1 , wherein a simultaneous localization and mapping (SLAM) process provides correction data, wherein:
corrections having a scale above a threshold are treated using a first correction process; and (e.g., large, global transformations)
corrections having a scale below the threshold are treated using a second correction process different than the first correction process.
19 . The method of claim 1 further comprising:
receiving correction data corresponding to a correction associated with device poses associated with the frames;
determining a global transformation and a change to non-rigid component based on the correction data; and
updating a 3D representation by:
applying a transform based on the global transformation to update the 3D representation; and
regenerating a subset of the 3D representation based on the change to the non-rigid component.
20 . The method of claim 1 , wherein the 3D representation is generated based on:
identifying previously-obtained key frames associated with the physical environment;
comparing the previously-obtained key frames with the plurality of frames of sensor data; and
determining whether to use one or more of the previously-obtained key frames to generate the 3D representation based on the comparing.
21 . A system comprising:
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising:
obtaining a plurality of frames of sensor data of a physical environment, the frames of sensor data captured while the device is moved along a path of multiple positions and multiple orientations within the physical environment;
identifying a first frame of the plurality of frames as a key frame based on a new key frame selection criterion, wherein the new key frame selection criterion assesses similarity between an evaluated frame and one or more previously-identified key frames;
determining to fuse a second frame of the plurality of frames with the key frame based on the second frame not being identified as another key frame based on the new key frame selection criterion;
fusing the second frame of the plurality of frames with the key frame, wherein the key frame aggregates data from the first frame and the second frame; and
generating a 3D representation of the physical environment based on the key frame.