Voxel meshing based on occupancy
Various implementations generate geometric representations based on depth information detected in physical environments. For example, an example process may include acquiring sensor data of an object in a physical environment, the sensor data including images of a physical environment captured via a camera on the device. The process may further include generating a three-dimensional (3D) voxel representation of the object based on the images. The process may further include determining signed distance values (SDVs) of voxel corners for voxels of the 3D voxel representation, the SDVs representing distances to surfaces of the object; determining occupancy data for the voxels of the 3D voxel representation, the occupancy data corresponding to whether the voxels are occupied by the object. The process may further include generating a 3D mesh of the object based on the SDVs of the voxel corners and the occupancy data for the 3D voxel representation.
1 . A method comprising:
at a device having a processor:
acquiring sensor data of an object in a physical environment, the sensor data comprising images of a physical environment captured via a camera on the device;
generating a three-dimensional (3D) voxel representation of the object based on the images;
determining signed distance values (SDVs) of voxel corners for voxels of the 3D voxel representation, the SDVs representing distances to surfaces of the object;
determining occupancy data for the voxels of the 3D voxel representation, the occupancy data corresponding to whether the voxels are occupied by the object; and
generating a 3D mesh of the object based on the SDVs of the voxel corners and the occupancy data for the 3D voxel representation.
2 . The method of claim 1 , wherein generating the 3D voxel representation of the object comprises generating a 3D point cloud of the object based on the sensor data, and generating the 3D voxel representation based on the 3D point cloud.
3 . The method of claim 1 , wherein determining the occupancy data for the voxels of the 3D voxel representation to determine whether the voxels are occupied by the object comprises identifying a likelihood that each of the voxels is at least partially occupied by a portion of the object.
4 . The method of claim 3 , wherein determining whether the voxels are occupied by the object comprises comparing the likelihood that each of the voxels is at least partially occupied by a portion of the object to an occupancy threshold.
5 . The method of claim 1 , wherein generating the 3D mesh of the object comprises:
determining whether a first voxel of the voxels of the 3D voxel representation is occupied by a portion of the object based on the occupancy data; and
in response determining that the first voxel is occupied, determining that the first voxel will include a mesh segment.
6 . The method of claim 1 , wherein generating the 3D mesh of the object comprises:
determining whether a first voxel of the voxels of the 3D voxel representation is occupied by a portion of the object based on the occupancy data; and
in response determining that the first voxel is occupied, determining that the first voxel will include a front mesh segment and a rear mesh segment together representing a thin portion of the object.
7 . The method of claim 1 , further comprising determining a mesh segment for a first voxel, the mesh segment corresponding to a surface of the object and determined using multiple SDVs for a voxel corner of the first voxel, the multiple SDVs corresponding to sensor data from different viewpoints.
8 . The method of claim 1 , further comprising determining a mesh segment for a first voxel, the mesh segment corresponding to a surface of the object and determined by:
identifying a subset of SDVs associated with sensor data obtained from viewpoints on a side of the object; and
generating the mesh segment based on the subset of SDVs.
9 . The method of claim 1 , further comprising determining a mesh segment for a first voxel, the mesh segment corresponding to a surface of the object and determined by:
identifying a first subset of SDVs associated with sensor data obtained from a first set of viewpoints on a first side of the object;
generating a first mesh segment based on the first subset of SDVs;
identifying a second subset of SDVs associated with sensor data obtained from a second set of viewpoints on a second side of the object; and
generating a second mesh segment based on the second subset of SDVs.
10 . The method of claim 1 , further comprising determining a mesh segment for a first voxel, the mesh segment corresponding to a surface of the object and determined by:
providing SDVs for voxel corners of the first voxel to a marching cubes-type algorithm, the provided SDVs comprising multiple SDVs for at least one of the voxel corners of the first voxel, the multiple SDVs corresponding to sensor data from different viewpoints; and
generating the mesh segment via the marching cubes-type algorithms, wherein the marching cubes-type algorithm accounts for the multiple SDVs in generating the mesh segment for the first voxel.
11 . The method of claim 1 , further comprising determining a mesh segment for a first voxel, the mesh segment corresponding to a surface of the object and determined by:
providing SDVs for voxel corners of the first voxel to a marching cubes-type algorithm, the provided SDVs comprising multiple SDVs for at least one of the voxel corners of the first voxel, the multiple SDVs corresponding to sensor data from different viewpoints;
generating a first mesh segment via the marching cubes-type algorithms, wherein the marching cubes-type algorithm accounts for the SDVs associated with viewpoints on one side of a thin portion of the object in generating the first mesh segment for the first voxel; and
generating a second mesh segment via the marching cubes-type algorithms, wherein the marching cubes-type algorithm accounts for the SDVs associated with viewpoints on another side of the thin portion of the object in generating the second mesh segment for the first voxel.
12 . The method of claim 1 , wherein the sensor data comprises depth data that is obtained using one or more depth cameras, wherein the depth data comprises pixel depth values from a viewpoint and a sensor position.
13 . A device comprising:
one or more sensors;
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations comprising:
acquiring sensor data of an object in a physical environment, the sensor data comprising images of a physical environment captured via a camera on the device;
generating a three-dimensional (3D) voxel representation of the object based on the images;
determining signed distance values (SDVs) of voxel corners for voxels of the 3D voxel representation, the SDVs representing distances to surfaces of the object;
determining occupancy data for the voxels of the 3D voxel representation, the occupancy data corresponding to whether the voxels are occupied by the object; and
generating a 3D mesh of the object based on the SDVs of the voxel corners and the occupancy data for the 3D voxel representation.
14 . The device of claim 13 , wherein generating the 3D voxel representation of the object comprises generating a 3D point cloud of the object based on the sensor data, and generating the 3D voxel representation based on the 3D point cloud.
15 . The device of claim 13 , wherein determining the occupancy data for the voxels of the 3D voxel representation to determine whether the voxels are occupied by the object comprises identifying a likelihood that each of the voxels is at least partially occupied by a portion of the object.
16 . The device of claim 15 , wherein determining whether the voxels are occupied by the object comprises comparing the likelihood that each of the voxels is at least partially occupied by a portion of the object to an occupancy threshold.
17 . The device of claim 13 , wherein generating the 3D mesh of the object comprises:
determining whether a first voxel of the voxels of the 3D voxel representation is occupied by a portion of the object based on the occupancy data; and
in response determining that the first voxel is occupied, determining that the first voxel will include a mesh segment.
18 . The device of claim 13 , wherein generating the 3D mesh of the object comprises:
determining whether a first voxel of the voxels of the 3D voxel representation is occupied by a portion of the object based on the occupancy data; and
in response determining that the first voxel is occupied, determining that the first voxel will include a front mesh segment and a rear mesh segment together representing a thin portion of the object.
19 . The device of claim 13 , wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, further cause the one or more processors to perform operations comprising determining a mesh segment for a first voxel, the mesh segment corresponding to a surface of the object and determined using multiple SDVs for a voxel corner of the first voxel, the multiple SDVs corresponding to sensor data from different viewpoints.
20 . A non-transitory computer-readable storage medium, storing program instructions executable on a device to perform operations comprising:
acquiring sensor data of an object in a physical environment, the sensor data comprising images of a physical environment captured via a camera on the device;
generating a three-dimensional (3D) voxel representation of the object based on the images;
determining signed distance values (SDVs) of voxel corners for voxels of the 3D voxel representation, the SDVs representing distances to surfaces of the object; determining occupancy data for the voxels of the 3D voxel representation, the occupancy data corresponding to whether the voxels are occupied by the object; and
generating a 3D mesh of the object based on the SDVs of the voxel corners and the occupancy data for the 3D voxel representation.