Similarity detection of three-dimensional object models
Implementations relate to methods, systems, and computer-readable media for performing similarity analysis of three-dimensional (3D) objects. In some implementations, the method includes obtaining a 3D model of the object that includes a mesh and a texture, encoding the mesh into a mesh feature vector using a first neural network, encoding the texture into a texture feature vector using a second neural network, computing a mesh distance between the mesh feature vector and a reference mesh feature vector of a reference object, computing a texture distance between the texture feature vector and a reference texture feature vector of the reference object, determining, based on the mesh distance and the texture distance, whether the object matches the reference object, classifying the object as a dissimilar object if the object does not match the reference object, and classifying the object as a similar object if the object matches the reference object.
1 . A computer-implemented method, comprising:
obtaining a three-dimensional (3D) model of an object, wherein the 3D model includes a mesh comprising a plurality of points each associated with a respective 3D coordinate and a texture, wherein the texture is applied to the mesh to render the object for display;
encoding the mesh into a mesh feature vector using a first neural network;
encoding the texture into a texture feature vector using a second neural network;
computing a mesh distance between the mesh feature vector and a reference mesh feature vector of a reference object;
computing a texture distance between the texture feature vector and a reference texture feature vector of the reference object;
determining, based on the mesh distance and the texture distance, whether the object matches the reference object;
classifying the object as a dissimilar object if the object does not match the reference object; and
classifying the object as a similar object if the object matches the reference object.
2 . The computer-implemented method of claim 1 , wherein the first neural network includes an input layer, a plurality of convolution layers, and one or more fully-connected (FC) layers, wherein encoding the mesh comprises providing the mesh as input at the input layer, and wherein the mesh feature vector is obtained as output of a final FC layer of the one or more FC layers.
3 . The computer-implemented method of claim 2 , wherein providing the mesh as input at the input layer comprises providing the 3D coordinates of the plurality of points to the input layer and wherein a number of nodes of the input layer is equal to a number of the plurality of points.
4 . The computer-implemented method of claim 1 , wherein the mesh feature vector is agnostic to operations that do not alter a shape of the mesh.
5 . The computer-implemented method of claim 4 , wherein the operations comprise one or more of: triangulation, resolution, rotation, translation, scale, subdivision, decimation, and combinations thereof.
6 . The computer-implemented method of claim 1 , wherein the texture is a two-dimensional (2D) image, and wherein the second neural network comprises a plurality of layers, and wherein the texture feature vector is obtained from a pre-final layer of the plurality of layers, and wherein a final layer of the second neural network is configured to output a classification for the texture.
7 . The computer-implemented method of claim 1 , wherein the first neural network comprises a mesh classifier, the method further comprising training the mesh classifier, wherein training the mesh classifier comprises:
providing, as input to the mesh classifier, training data that includes a plurality of reference meshes of training objects and a respective set of labels of different classes associated with each training object;
for each training object:
generating, based on the mesh of the training object and by the mesh classifier, a respective mesh feature vector;
generating, based on the mesh feature vector, one or more predicted labels for the training object; and
computing a loss value based on comparison of the one or more predicted labels with the respective set of labels associated with the training object; and
adjusting one or more parameters of the mesh classifier based on the loss values.
8 . The computer-implemented method of claim 1 , wherein determining whether the object matches the reference object comprises calculating an aggregate score based on the mesh distance and the texture distance.
9 . The computer-implemented method of claim 1 , wherein the object is classified as the similar object, and wherein the method further comprises comparing a storage footprint of the object and the reference object, and storing, on a storage device of a virtual experience platform, one of the object and the reference object, based on their respective storage footprint.
10 . The computer-implemented method of claim 1 , wherein the object is received from a user account of a virtual experience platform, and wherein subsequent to classifying the object as the similar object, the method further comprises transmitting a message to a device associated with the user account, wherein the message is indicative of similarity of the object to the reference object.
11 . A non-transitory computer-readable medium with instructions stored thereon that, responsive to execution by a processing device, cause the processing device to perform operations comprising:
obtaining a three-dimensional (3D) model of an object, wherein the 3D model includes a mesh comprising a plurality of points each associated with a respective 3D coordinate and a texture, wherein the texture is applied to the mesh to render the object for display;
encoding the mesh into a mesh feature vector using a first neural network;
encoding the texture into a texture feature vector using a second neural network;
computing a mesh distance between the mesh feature vector and a reference mesh feature vector of a reference object;
computing a texture distance between the texture feature vector and a reference texture feature vector of the reference object;
determining, based on the mesh distance and the texture distance, whether the object matches the reference object;
classifying the object as a dissimilar object if the object does not match the reference object; and
classifying the object as a similar object if the object matches the reference object.
12 . The non-transitory computer-readable medium of claim 11 , wherein the first neural network includes an input layer, a plurality of convolution layers, and one or more fully-connected (FC) layers, wherein encoding the mesh comprises providing the mesh as input at the input layer, and wherein the mesh feature vector is obtained as output of a final FC layer of the one or more FC layers.
13 . The non-transitory computer-readable medium of claim 12 , wherein providing the mesh as input at the input layer comprises providing the 3D coordinates of the plurality of points to the input layer, and wherein a number of nodes of the input layer is equal to a number of the plurality of points.
14 . The non-transitory computer-readable medium of claim 11 , wherein the object is received from a user account of a virtual experience platform, and wherein subsequent to classifying the object as the similar object, the operations further comprise transmitting a message to a device associated with the user account, wherein the message is indicative of similarity of the object to the reference object.
15 . The non-transitory computer-readable medium of claim 11 , wherein the texture is a two-dimensional (2D) image, and wherein the second neural network comprises a plurality of layers, and wherein the texture feature vector is obtained from a pre-final layer of the plurality of layers, and wherein a final layer of the second neural network is configured to output a classification for the texture.
16 . A system comprising:
a memory with instructions stored thereon; and
a processing device, coupled to the memory, the processing device configured to access the memory and execute the instructions, wherein the instructions cause the processing device to perform operations comprising:
obtaining a three-dimensional (3D) model of an object, wherein the 3D model includes a mesh comprising a plurality of points each associated with a respective 3D coordinate and a texture, wherein the texture is applied to the mesh to render the object for display;
encoding the mesh into a mesh feature vector using a first neural network;
encoding the texture into a texture feature vector using a second neural network;
computing a mesh distance between the mesh feature vector and a reference mesh feature vector of a reference object;
computing a texture distance between the texture feature vector and a reference texture feature vector of the reference object;
determining, based on the mesh distance and the texture distance, whether the object matches the reference object;
classifying the object as a dissimilar object if the object does not match the reference object; and
classifying the object as a similar object if the object matches the reference object.
17 . The system of claim 16 , wherein the first neural network includes an input layer, a plurality of convolution layers, and one or more fully-connected (FC) layers, wherein encoding the mesh comprises providing the mesh as input at the input layer, and wherein the mesh feature vector is obtained as output of a final FC layer of the one or more FC layers.
18 . The system of claim 17 , wherein providing the mesh as input at the input layer comprises providing the 3D coordinates of the plurality of points to the input layer, and wherein a number of nodes of the input layer is equal to a number of the plurality of points.
19 . The system of claim 16 , wherein the object is classified as the similar object, and wherein the operations further comprise comparing a storage footprint of the object and the reference object, and storing, on a storage device of a virtual experience platform, one of the object and the reference object, based on their respective storage footprint.
20 . The system of claim 16 , wherein the texture is a two-dimensional (2D) image, and wherein the second neural network comprises a plurality of layers, and wherein the texture feature vector is obtained from a pre-final layer of the plurality of layers, and wherein a final layer of the second neural network is configured to output a classification for the texture.