IP Library › Granted Patent US 12,367,233
Granted Patent B2
US 12,367,233 · App. 18/233,181 · Granted Jul 22, 2025

Determining 3D models corresponding to an image

Inventors: Robert Banfield (Riverview, FL); Aristodimos Komninos (Athens, GR); Jacques Harvent (Le Pereux-sur-Marne, FR); Michael Tadros (Boulder, CO); Karolina Torttila (Helsinki, FI)
Assignee: Trimble Inc.
G06F16/532G06F16/538G06F16/56G06T15/10G06T17/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,367,233
App. No.
18/233,181
Granted
Jul 22, 2025
Kind
B2
Abstract

One or more three-dimensional models that corresponds to at least one two-dimensional image are determined by receiving image data that corresponds to the at least one two-dimensional image, generating features based on the image data corresponding to the at least one two-dimensional image, generating a representation vector for the at least one two-dimensional image by transforming the features into a predetermined amount of numerical representations corresponding to the features, and outputting the representation vector for the at least one two-dimensional image to facilitate a search query for the one or more three-dimensional models associated with the at least one two-dimensional image.

Claims (78)

1. A method comprising:

receiving, by a computing device, image data that corresponds to at least one two-dimensional image;

generating, by the computing device and using supervised layers of a hybrid machine-learning model, features based on the image data corresponding to the at least one two-dimensional image;

generating, by the computing device and by using an unsupervised layer of the hybrid machine-learning model, a representation vector for the at least one two-dimensional image by transforming the features into a predetermined amount of numerical representations corresponding to the features; and

outputting, by the computing device, the representation vector for the at least one two-dimensional image to facilitate a search query for one or more three-dimensional models associated with the at least one two-dimensional image;

wherein the hybrid machine-learning model has an architecture such that each of the supervised layers precedes the unsupervised layer and the supervised layers generate an input for the unsupervised layer, each of the supervised layers having been trained via a supervised training technique and the unsupervised layer having been trained via an unsupervised training technique.

2. The method of claim 1 , further comprising:

receiving, by the computing device, model data that includes data representing a set of three-dimensional models;

generating, by the computing device, a set of second representation vectors, for each three-dimensional model included in the set of three-dimensional models:

generating, by the computing device, a two-dimensional snapshot based on a particular view of the three-dimensional model;

generating, by the computing device and using the supervised layers of the hybrid machine-learning model, features corresponding to a subset of the model data, wherein the subset corresponds to the two-dimensional snapshot; and

generating, by the computing device and using the unsupervised layer of the hybrid machine-learning model, a second representation vector based on the features corresponding to the subset of the model data, wherein the second representation vector represents the two-dimensional snapshot; and

storing each second representation vector of the set of second representation vectors in a model query data store.

3. The method of claim 2 , wherein:

receiving the image data comprises receiving, by the computing device, a model search query that indicates that an entity is searching for a three-dimensional model based on the at least one two-dimensional image; and

outputting the representation vector comprises comparing, by the computing device and in response to receiving the model search query, the representation vector to each second representation vector of the set of second representation vectors stored in the model query data store.

4. The method of claim 3 , further comprising:

determining, by the computing device and based on comparing the representation vector to each second representation vector, that a particular second representation vector of the set of second representation vectors is most similar to the representation vector; and

outputting, by the computing device, a particular three-dimensional model of the one or more three-dimensional models, wherein the particular second representation vector corresponds to the particular three-dimensional model.

5. The method of claim 1 , wherein receiving the image data includes:

receiving, by the computing device, a three-dimensional model; and

generating, by the computing device, the at least one two-dimensional image based on the three-dimensional model.

6. The method of claim 5 , wherein generating the at least one two-dimensional image comprises generating, by the computing device, the at least one two-dimensional image by extracting the image data from a two-dimensional snapshot of the three-dimensional model.

7. The method of claim 6 , further comprising:

determining, by the computing device and based on comparing the representation vector to each second representation vector of a set of second representation vectors corresponding to the one or more three-dimensional models, that a particular second representation vector of the set of second representation vectors is most similar to the representation vector; and

outputting, by the computing device, a particular three-dimensional model of the one or more three-dimensional models, wherein the particular second representation vector corresponds to the particular three-dimensional model.

8. The method of claim 7 , wherein generating the at least one two-dimensional image, determining that the particular second representation vector of the set of second representation vectors is most similar to the representation vector, and outputting the particular three-dimensional model are performed substantially contemporaneously.

9. A system comprising:

a processing device; and

a non-transitory computer-readable medium comprising instructions executable by the processing device to cause the processing device to perform operations comprising:

receiving image data that corresponds to at least one two-dimensional image;

generating, using supervised layers of a hybrid machine-learning model, features based on the image data corresponding to the at least one two-dimensional image;

generating, by using an unsupervised layer of the hybrid machine-learning model, a representation vector for the at least one two-dimensional image by transforming the features into a predetermined amount of numerical representations corresponding to the features; and

outputting the representation vector for the at least one two-dimensional image to facilitate a search query for one or more three-dimensional models associated with the at least one two-dimensional image;

wherein the hybrid machine-learning model has an architecture such that each of the supervised layers precedes the unsupervised layer and the supervised layers generate an input for the unsupervised layer, each of the supervised layers having been trained via a supervised training technique and the unsupervised layer having been trained via an unsupervised training technique.

10. The system of claim 9 , wherein the operation of receiving the image data includes:

receiving a three-dimensional model; and

generating the at least one two-dimensional image based on the three-dimensional model.

11. The system of claim 10 , wherein the operation of generating the at least one two-dimensional image comprises generating the at least one two-dimensional image by extracting the image data from a two-dimensional snapshot of the three-dimensional model.

12. The system of claim 11 , wherein:

the operation of receiving the image data comprises receiving a model search query that indicates that an entity is searching for the three-dimensional model; and

the operation of outputting the representation vector comprises comparing, in response to receiving the model search query, the representation vector to each second representation vector of a set of second representation vectors corresponding to a set of three-dimensional models.

13. The system of claim 12 , further comprising:

receiving model data that includes the set of three-dimensional models;

generating the set of second representation vectors by, for each three-dimensional model included in the set of three-dimensional models:

generating a two-dimensional image based on a particular view of the three-dimensional model;

generating, using the supervised layers of the hybrid machine-learning model, features corresponding to a subset of the model data corresponding to the two-dimensional image; and

generating, using the unsupervised layer of the hybrid machine-learning model, a second corresponding representation vector based on the features corresponding to the subset of the model data, wherein the second corresponding representation vector represents the two-dimensional image; and

storing each second representation vector of the set of second representation vectors in a model query data store.

14. The system of claim 13 , wherein the operations further comprise:

determining, based on comparing the representation vector to each second representation vector of the set of second representation vectors, that a particular second representation vector of the set of second representation vectors is most similar to the representation vector; and

outputting a particular three-dimensional model of the one or more three-dimensional models, wherein the particular second representation vector corresponds to the particular three-dimensional model; and

wherein the operation of generating the at least one two-dimensional image, the operation of determining that the particular second representation vector of the set of second representation vectors is most similar to the representation vector, and the operation of outputting the particular three-dimensional model are performed substantially contemporaneously.

15. A non-transitory computer-readable medium comprising instructions executable by a processing device to cause the processing device to perform operations comprising:

receiving image data that corresponds to at least one two-dimensional image;

generating, using supervised layers of a hybrid machine-learning model, features based on the image data corresponding to the at least one two-dimensional image;

generating, by using an unsupervised layer of the hybrid machine-learning model, a representation vector for the at least one two-dimensional image by transforming the features into a predetermined amount of numerical representations corresponding to the features; and

outputting the representation vector for the at least one two-dimensional image to facilitate a search query for one or more three-dimensional models associated with the at least one two-dimensional image;

wherein the hybrid machine-learning model has an architecture such that each of the supervised layers precedes the unsupervised layer and the supervised layers generate an input for the unsupervised layer, each of the supervised layers having been trained via a supervised training technique and the unsupervised layer having been trained via an unsupervised training technique.

16. The non-transitory computer-readable medium of claim 15 , wherein the operations further comprise:

receiving model data that includes data representing a set of three-dimensional models;

generating a set of second representation vectors by, for each three-dimensional model included in the set of three-dimensional models:

generating a two-dimensional snapshot based on a particular view of the three-dimensional model;

generating, using the supervised layers of the hybrid machine-learning model, features corresponding to a subset of the model data, wherein the subset corresponds to the two-dimensional snapshot; and

generating, using the unsupervised layer of the hybrid machine-learning model, a second representation vector based on the features corresponding to the subset of the model data, wherein the second representation vector represents the two-dimensional snapshot; and

storing each second representation vector of the set of second representation vectors in a model query data store.

17. The non-transitory computer-readable medium of claim 16 , wherein:

the operation of receiving the image data comprises receiving a model search query that indicates that an entity is searching for a three-dimensional model based on the at least one two-dimensional image; and

the operation of outputting the representation vector comprises comparing, in response to receiving the model search query, the representation vector to each second representation vector of the set of second representation vectors stored in the model query data store.

18. The non-transitory computer-readable medium of claim 17 , wherein the operations further comprise:

determining, based on comparing the representation vector to each second representation vector, that a particular second representation vector of the set of second representation vectors is most similar to the representation vector; and

outputting a particular three-dimensional model of the one or more three-dimensional models, wherein the particular second representation vector corresponds to the particular three-dimensional model.

19. The non-transitory computer-readable medium of claim 15 , wherein the operation of receiving the image data includes:

receiving a three-dimensional model; and

generating the at least one two-dimensional image based on the three-dimensional model.

20. The non-transitory computer-readable medium of claim 15 , wherein the operation of generating the at least one two-dimensional image comprises generating the at least one two-dimensional image by extracting the image data from a two-dimensional rendering of a three-dimensional model, and wherein the operations further comprise:

determining, based on comparing the representation vector to each second representation vector of a set of second representation vectors corresponding to the one or more three-dimensional models, that a particular second representation vector of the set of second representation vectors is most similar to the representation vector; and

outputting a particular three-dimensional model of the one or more three-dimensional models, wherein the particular second representation vector corresponds to the particular three-dimensional model.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 11, 2023
From: BANFIELD, ROBERT; KOMNINOS, ARISTODIMOS; HARVENT, JACQUES; TADROS, MICHAEL; TORTTILA, KAROLIINA
To: TRIMBLE INC.
Reel/Frame 064570/0474 →
Continuity (1)
Related Publication 20240104132A1 · Mar 28, 2024
References Cited (5)
US 20190147221A1 · Grabner · 2019 [cited by examiner]
US 20210243362A1 · Castillo · 2021 [cited by examiner]
US 20240054731A1 · Wolke · 2024 [cited by examiner]
Extended European Search Report for Application No. 23198733.0-1203, mailed Jan. 23, 2024, 9 pages. [cited by applicant]
Singh, A. et al., “Scatternet Hybrid Deep Learning (SHDL) Network for Object Classification,” 2017 IEEE International Workshop on Machine Learning for Signal Processing, Sep. 25-28, 2017, Tokyo, Japan, 6 pages. [cited by applicant]