Fitting 3D primitives to a high-resolution point cloud
Embodiments provide systems, methods, and computer storage media for fitting 3D primitives to a 3D point cloud. In an example embodiment, 3D primitives are fit to a 3D point cloud using a global primitive fitting network that evaluates the entire 3D point cloud and a local primitive fitting network that evaluates local patches of the 3D point cloud. The global primitive fitting network regresses a representation of larger (global) primitives that fit the global structure. To identify smaller 3D primitives for regions with fine detail, local patches are constructed by sampling from a pool of points likely to contain fine detail, and the local primitive fitting network regresses a representation of smaller (local) primitives that fit the local structure of each of the local patches. The global and local primitives are merged into a combined, multi-scale set of fitted primitives, and representative primitive parameters are computed for each fitted primitive.
1. One or more computer storage media storing computer-useable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations comprising:
generating, by a global primitive fitting network comprising a first instance of a base architecture, a representation of global 3D primitives that approximate a global structure of a three-dimensional (3D) point cloud;
generating, by a local primitive fitting network comprising a second instance of the base architecture, a representation of local 3D primitives that approximate a local structure of a local patch of the 3D point cloud; and
generating a representation of a set of fitted 3D primitives for the 3D point cloud based on the representation of the global 3D primitives and the representation of the local 3D primitives.
2. The one or more computer storage media of claim 1 , wherein the local primitive fitting network is configured to predict the representation of the local 3D primitive s from a first input representation of 3D points of the local patch and a second input representation of global contextual information generated by the global primitive fitting network.
3. The one or more computer storage media of claim 1 , wherein the local patch includes a seed point, wherein the global primitive fitting network is configured to encode the 3D point cloud into an object latent vector and encode the seed point into a seed point feature vector, wherein the local primitive fitting network is configured to encode an input representation of 3D points of the local patch into a patch latent vector and decode a combined representation of the object latent vector, the seed point feature vector, and the patch latent vector into the representation of the local 3D primitives.
4. The one or more computer storage media of claim 1 , the operations further comprising:
identifying, using a fine detail identification network, a pool of 3D points in the 3D point cloud predicted to represent fine detail; and
generating the local patch using the pool of 3D points.
5. The one or more computer storage media of claim 1 , the operations further comprising:
downsampling the 3D point cloud to generate a downsampled 3D point cloud;
sampling a query point from the downsampled 3D point cloud; and
generating the local patch by grouping a set of nearest neighbors of the query point from a full resolution of the 3D point cloud.
6. The one or more computer storage media of claim 1 , wherein generating the representation of the set of fitted 3D primitives comprises aggregating and de-duplicating the global 3D primitives and the local 3D primitives based on maximizing intersections of points that are common to initial primitives that are assigned to a common one of the fitted 3D primitives.
7. The one or more computer storage media of claim 1 , wherein generating the representation of the set of fitted 3D primitives comprises merging the global 3D primitives and the local 3D primitives and preventing merging of any of the local 3D primitives generated from the local patch of the 3D point cloud with another one of the local 3D primitives generated from the local patch.
8. The one or more computer storage media of claim 1 , the operations further comprising identifying a first primitive type for a particular primitive in the set of fitted 3D primitives, the particular primitive having assigned points from the 3D point cloud, based on summing probabilities, generated by the global primitive fitting network and the local primitive fitting network, that each of the assigned points belongs to the first primitive type.
9. The one or more computer storage media of claim 1 , wherein the representation of the set of fitted 3D primitives comprises, for at least one point in the 3D point cloud, a surface normal for the at least one point generated based on averaging a first surface normal generated for the at least one point by the global primitive fitting network and one or more surface normals generated for the at least one point by the local primitive fitting network.
10. A computerized method comprising:
generating, by a global primitive fitting network, a representation of global 3D primitives based on a global structure of a three-dimensional (3D) point cloud;
generating, by a local primitive fitting network, a representation of local 3D primitives based on a local structure of a local patch generated using a pool of 3D points of the 3D point cloud predicted to represent fine detail; and
generating a representation of a set of fitted 3D primitives for the 3D point cloud based on merging the global 3D primitives and the local 3D primitives.
11. The computerized method of claim 10 , wherein the local primitive fitting network is configured to predict the representation of the local 3D primitives from a first input representation of 3D points of the local patch and a second input representation of global contextual information generated by the global primitive fitting network.
12. The computerized method of claim 10 , wherein the local patch includes a seed point, wherein the global primitive fitting network is configured to encode the 3D point cloud into an object latent vector and encode the seed point into a seed point feature vector, wherein the local primitive fitting network is configured to encode an input representation of 3D points of the local patch into a patch latent vector and decode a combined representation of the object latent vector, the seed point feature vector, and the patch latent vector into the representation of the local 3D primitives.
13. The computerized method of claim 10 , further comprising:
downsampling the 3D point cloud to generate a downsampled 3D point cloud;
sampling a query point from the downsampled 3D point cloud; and
generating the local patch by grouping a set of nearest neighbors of the query point from a full resolution of the 3D point cloud.
14. The computerized method of claim 10 , wherein generating the representation of the set of fitted 3D primitives comprises aggregating and de-duplicating the global 3D primitives and the local 3D primitives based on maximizing intersections of points that are common to initial primitives that are assigned to a common one of the fitted 3D primitives.
15. The computerized method of claim 10 , wherein generating the representation of the set of fitted 3D primitives comprises merging the global 3D primitives and the local 3D primitives and preventing merging of any of the local 3D primitives generated from the local patch of the 3D point cloud with another one of the local 3D primitives generated from the local patch.
16. The computerized method of claim 10 , further comprising identifying a first primitive type for a particular primitive in the set of fitted 3D primitives, the particular primitive having assigned points from the 3D point cloud, based on summing probabilities, generated by the global primitive fitting network and the local primitive fitting network, that each of the assigned points belongs to the first primitive type.
17. The computerized method of claim 10 , wherein the representation of the set of fitted 3D primitives comprises, for at least one point in the 3D point cloud, a surface normal for the at least one point generated based on averaging a first surface normal generated for the at least one point by the global primitive fitting network and one or more surface normals generated for the at least one point by the local primitive fitting network.
18. A computer system comprising:
one or more hardware processors and memory configured to provide computer program instructions, that, when used by the one or more hardware processors, cause the one or more hardware processors to perform operations comprising:
generating, by a global primitive fitting network, a representation of global 3D primitives corresponding to a global structure of a three-dimensional (3D) point cloud;
generating, by a local primitive fitting network based at least on global contextual information generated by the global primitive fitting network, a representation of local 3D primitives corresponding to a local structure of a local patch of the 3D point cloud; and
generating a representation of a set of fitted 3D primitives for the 3D point cloud based on aggregating and de-duplicating the global 3D primitives and the local 3D primitives.
19. The computer system of claim 18 , the operations further comprising identifying a first primitive type for a particular primitive in the set of fitted 3D primitives, the particular primitive having assigned points from the 3D point cloud, based on summing probabilities, generated by the global primitive fitting network and the local primitive fitting network, that each of the assigned points belongs to the first primitive type.
20. The computer system of claim 18 , wherein the representation of the set of fitted 3D primitives comprises, for at least one point in the 3D point cloud, a surface normal for the at least one point generated based on averaging a first surface normal generated for the at least one point by the global primitive fitting network and one or more surface normals generated for the at least one point by the local primitive fitting network.