Selecting representative image views for 3D object models in synthetic content creation systems and applications
Approaches presented herein provide for automatic generation of representative two-dimensional (2D) images for three-dimensional (3D) objects or assets. In generating these 2D images, a set of options is determined such as may relate to viewpoint or other parameters of a virtual camera. A set of sample points is determined from which to generate 2D images of a 3D model, for example, with 2D images being processed using a classifier to determine which of these images generates a classification with highest confidence or probability, individually or with respect to other classifications. The sample point for this selected image can then be used to select nearby sample points as part of a refinement or optimization process, where 2D images can again be generated and processed using a classifier to identify a 2D image with highest classification probability or confidence, which can be selected as representative of the 3D object or asset.
1 . A method, comprising:
generating a plurality of two-dimensional (2D) images corresponding to different views of a three-dimensional (3D) representation of an object;
providing the plurality of 2D images as input to a machine learning model;
inferring, by the machine learning model using the input, respective probabilities of the object belonging to each object class of a plurality of object classes; and
selecting, as representative of the object, a 2D image of the plurality of 2D images that corresponds to a highest probability of the respective probabilities with an object class that matches a class of the 3D representation of the object and that corresponds, relative to at least one other 2D image of the plurality of 2D images, to a lower probability of the respective probabilities with at least one other object class.
2 . The method of claim 1 , further comprising:
determining a bounding geometry surrounding at least a portion of the 3D representation of the object as depicted in the 2D image selected as representative of the object; and
selecting one or more first sample points, on or adjacent the bounding geometry, from which to render one or more additional 2D images of the 3D representation of the object.
3 . The method of claim 2 , wherein at least one sample point of the one or more first sample points is selected at random, at one or more fixed intervals, or according to a sampling algorithm.
4 . The method of claim 2 , further comprising:
generating the one or more additional 2D images corresponding to the one or more first sample points;
providing the additional 2D images as input to the machine learning model to determine, for the additional 2D images, respective probabilities of the object belong to an object class as depicted in the additional 2D images; and
selecting, from the additional 2D images, at least one additional 2D image having the highest probability of the object belonging to the object class as representative of the object.
5 . The method of claim 1 , further comprising:
generating the plurality of 2D images and selecting the 2D image, representative of the object, automatically and without human input.
6 . The method of claim 1 , further comprising:
setting one or more values for one or more properties of at least one of a virtual camera or the 3D representation of the object before generating the plurality of 2D images and providing the 2D images as input to the classifier network.
7 . The method of claim 5 , wherein the one or more properties include at least one of a zoom level, a field of view, a camera distance, a lighting or an illumination property, a color property, a background property, a texture property, an object pose, or an object variant.
8 . The method of claim 1 , further comprising:
providing the selected 2D image as a 2D thumbnail image for a response to an at least partially image-based search for information corresponding to the object.
9 . A system, comprising:
one or more processing units to:
generate a plurality of two-dimensional (2D) images corresponding to different views of a three-dimensional (3D) model;
predict, using a machine learning model, respective confidence values for the plurality of 2D images that at least a portion of an instance of each 3D model of a plurality of 3D models is depicted; and
select, as representative of the 3D model, the generated 2D image that corresponds to a higher confidence value, than at least one other 2D image, of the respective confidence values with the 3D model and that corresponds to a lower confidence value, than at least one other 2D image, of the respective confidence values with at least one other 3D model.
10 . The system of claim 9 , wherein the one or more processing units are further to:
determine a bounding geometry surrounding at least a portion of the 3D model; and
select one or more first sample points, on or adjacent the bounding geometry, from which to render one or more additional 2D images.
11 . The system of claim 10 , wherein the one or more processing units are further to:
generate the one or more additional 2D images corresponding to the one or more first sample points;
predict, using the machine learning model, respective additional confidence values corresponding to the generated additional 2D images, an additional confidence value of the respective additional confidence values being representative of a probability that at least a portion of an instance of the 3D model is depicted in the respective additional 2D image; and
select, from the generated additional 2D images, at least one generated additional 2D image having the highest confidence value as representative of the 3D model.
12 . The system of claim 10 , wherein the one or more processing units are further to:
generate the plurality of 2D images and select the 2D image automatically and without human input.
13 . The system of claim 9 , wherein the one or more processing units are further to:
set one or more values for one or more properties of at least one of a virtual camera or the 3D model before generating the 2D images and predicting the respective confidence values using the machine learning model.
14 . The system of claim 13 , wherein the one or more properties include at least one of a zoom level, a field of view, a camera distance, a lighting or an illumination property, a color property, a background property, a texture property, an object pose, or an object variant.
15 . The system of claim 9 , wherein the system comprises at least one of:
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing simulation operations to test or validate autonomous machine applications;
a system for rendering graphical output;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system for generating synthetic data;
a system incorporating one or more Virtual Machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
16 . A processor comprising:
one or more processing units to select a first two-dimensional (2D) image from a plurality of 2D images corresponding to different views of a three-dimensional (3D) model, the first 2D image of the plurality of 2D images being selected based at least on having a higher probability of being classified by a machine learning model as a member of a class of units represented by the 3D model than an associated probability of at least one other 2D image of the plurality of 2D images and being selected based at least on having a lower probability of being classified by the machine learning model as a member of a class of units represented by another 3D model than an associated probability of at least one other 2D image of the plurality of 2D images.
17 . The processor of claim 16 , wherein the one or more processing units are further to:
determine a bounding geometry surrounding the 3D model; and
select a first plurality of sample points, on or adjacent the bounding geometry, from which to render one or more additional 2D images of the 3D model from one or more different views.
18 . The processor of claim 17 , wherein the one or more processing units are further to:
generate the one or more additional 2D images corresponding to the first plurality of sample points; and
select, from the one or more additional 2D images, at least one generated additional 2D image having the highest confidence value as representative of the 3D model.
19 . The processor of claim 16 , wherein the one or more processing units are further to:
generate the plurality of 2D images and select the first 2D image automatically and without human input.
20 . The processor of claim 18 , wherein the processor is comprised in at least one of:
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.