Planar splatting
Techniques are described for image processing. For example, a computing device can segment, using a first neural network, image(s) of a scene to determine respective segments for each of the image(s). The computing device can determine, using a second neural network, normal vectors for each of the image(s). The computing device can generate a graph based on each respective segment for each image, each respective normal vectors for each image, and estimated planar distances. The computing device can partition, based on the normal vectors and the estimated planar distances, the graph to determine indexes associated with Gaussian primitives. The computing device can assign, using linear regression, each descriptor of a plurality of descriptors to an index of the plurality of indexes based on a respective weight. The computing device can merge, using a Gaussian tree, Gaussian primitives of the Gaussian primitives with associated indexes that are similar to each other.
1 . An apparatus to process one or more images, the apparatus comprising:
one or more memories configured to store one or more images; and
one or more processors coupled to the one or more memories and configured to:
segment, using a first neural network, the one or more images of a scene to determine a respective plurality of segments for each image of the one or more images;
determine, using a second neural network, a respective plurality of normal vectors for each image of the one or more images, wherein each normal vector of the respective plurality of normal vectors is orthogonal to a surface associated with a respective primitive of a plurality of primitives associated with an image of the one or more images;
generate a graph based on each respective plurality of segments for each image, each respective plurality of normal vectors for each image, and a plurality of estimated planar distances;
partition, based on each respective plurality of normal vectors and the plurality of estimated planar distances, the graph to determine a plurality of indexes associated with the plurality of primitives;
assign each descriptor of a plurality of descriptors to an index of the plurality of indexes based on a respective weight; and
merge primitives of the plurality of primitives with associated indexes of the plurality of indexes that are similar to each other.
2 . The apparatus of claim 1 , wherein the first neural network is a segment anything model (SAM).
3 . The apparatus of claim 1 , wherein the second neural network is a normal neural network.
4 . The apparatus of claim 1 , wherein the graph is region adjacency graph (RAG).
5 . The apparatus of claim 1 , wherein each segment of the respective plurality of segments is represented as a node of a plurality of nodes in the graph.
6 . The apparatus of claim 5 , wherein each node of the plurality of nodes is connected by one or more edges to one or more adjacent nodes of the plurality of nodes.
7 . The apparatus of claim 6 , wherein, to partition the graph, the one or more processors are configured to remove at least one of the one or more edges.
8 . The apparatus of claim 1 , wherein the one or more processors are configured to update, using a recurrent-mean shift, the plurality of descriptors.
9 . The apparatus of claim 1 , wherein the plurality of primitives includes a plurality of Gaussian primitives, and wherein the respective primitive is a Gaussian primitive.
10 . The apparatus of claim 1 , wherein the one or more processors are configured to assign each descriptor to the index using linear regression.
11 . The apparatus of claim 1 , wherein the one or more processors are configured to merge the primitives of the plurality of primitives with the associated indexes using a binary tree.
12 . The apparatus of claim 11 , wherein the binary tree is a Gaussian tree.
13 . The apparatus of claim 1 , further comprising one or more cameras configured to capture the one or more images.
14 . A method of processing images, the method comprising:
segmenting, using a first neural network, one or more images of a scene to determine a respective plurality of segments for each image of the one or more images;
determining, using a second neural network, a respective plurality of normal vectors for each image of the one or more images, wherein each normal vector of the respective plurality of normal vectors is orthogonal to a surface associated with a respective primitive of a plurality of primitives associated with an image of the one or more images;
generating a graph based on each respective plurality of segments for each image, each respective plurality of normal vectors for each image, and a plurality of estimated planar distances;
partitioning, based on each respective plurality of normal vectors and the plurality of estimated planar distances, the graph to determine a plurality of indexes associated with the plurality of primitives;
assigning each descriptor of a plurality of descriptors to an index of the plurality of indexes based on a respective weight; and
merging primitives of the plurality of primitives with associated indexes of the plurality of indexes that are similar to each other.
15 . The method of claim 14 , wherein the first neural network is a segment anything model (SAM).
16 . The method of claim 14 , wherein the second neural network is a normal neural network.
17 . The method of claim 14 , wherein the plurality of primitives includes a plurality of Gaussian primitives, and wherein the respective primitive is a Gaussian primitive.
18 . The method of claim 14 , wherein each descriptor is to the index using linear regression.
19 . The method of claim 14 , wherein the primitives of the plurality of primitives are merged with the associated indexes using a binary tree.
20 . The method of claim 19 , wherein the binary tree is a Gaussian tree.