IP Library › Granted Patent US 12,293,459
Granted Patent B2
US 12,293,459 · App. 18/440,270 · Granted May 6, 2025

Technologies for 3D placement of virtual objects from a 2D layout

Inventors: Bret Besecker (St. Petersburg, FL); Barry Besecker (Xenia, OH); Jeffrey L. Cowgill, Jr. (Bellbrook, OH); Jonathan Jekeli (Dayton, OH)
Assignee: Marxent Labs LLC
G06T17/10G06Q30/0643G06T2200/04G06T2200/08
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,293,459
App. No.
18/440,270
Granted
May 6, 2025
Kind
B2
Abstract

Technologies for 3D virtual environment placement of 3D models based on 2D images are disclosed. At least an outline of a 3D virtual environment may be generated. A 2D image of one or more 2D images may be identified. A first product from the first 2D image may be identified. At least one 3D model of one or more 3D models based, at least, on the first product may be determined. A first location for placement of the first product in the 3D virtual environment may be identified. The at least one 3D model may be added within the 3D virtual environment based, at least, on the first location. The 3D virtual environment may be rendered into a visually interpretable form. A second product may be identified from the first 2D image, forming a first grouping of products. A starting element for the first grouping of products may be determined.

Claims (34)

1. A computer-implemented system for performing image analysis of a two-dimensional (2D) image to identify a first object in the 2D image and for determining an available product corresponding to the first object, the computer-implemented system comprising:

a resource database configured to store a data structure for storing a set of product data for a product comprising a set of attributes, a unique product identifier, and three-dimensional (3D) model data;

a processor;

a memory encoded with instructions which when executed, cause the processor to:

provide a first graphical user interface portion configured to present one or more images from a set of 2D images and a first user interface element to enable the user to select one of the 2D images;

operate an image analyzer component, upon receipt of a user-selected 2D image, to perform an image analysis process on the 2D image to identify the first object depicted in the 2D image;

determine the available product corresponding to the first object;

provide 3D model data for the available product corresponding to the first object to a rendering engine for rendering a 3D model of the available product in a 3D virtual environment; and

provide a second graphical user interface portion configured to display the 3D virtual environment and a second user interface element that enables the user to move the 3D model to a second location for placement in the 3D virtual environment.

2. The computer-implemented system of claim 1 , wherein the processor is further caused to:

determine a set of attributes associated with the first object, the set of attributes comprising at least one of object type, object function, object color, and/or object size.

3. The computer-implemented system of claim 2 , wherein the processor is further caused to:

determine the available product from the resource database that includes at least some of the set of attributes associated with the first object.

4. The computer-implemented system of claim 1 , wherein the processor is further caused to:

obtain, from the resource database, the 3D model data for the available product.

5. The computer-implemented system of claim 1 , wherein the set of attributes further comprises object texture and object composing material.

6. The computer-implemented system of claim 1 , wherein the resource database is split into a local resource database and a master resource database and further comprising a resource manager configured to receive a request for 3D model data for the available product and determine whether the requested 3D model data is stored in the local resource database and retrieve the 3D model data from the local resource database.

7. The computer-implemented system of claim 1 , wherein the resource database is split into a local resource database and a master resource database and further comprising a resource manager configured to receive a request for 3D model data for the available product and determine whether the requested 3D model data is stored in the local resource database and, if not, determine whether the requested 3D model data is stored in the master resource database and retrieve the 3D model data from the master resource database.

8. The computer-implemented system of claim 1 , wherein the second graphical user interface portion is further configured to display the 3D model data for a set of available products.

9. The computer-implemented system of claim 1 , wherein the set of 2D images comprise images of products and a layout of products in a room, wherein the image analyzer component is further configured to process the 2D image to extract from the 2D image an arrangement and/or decor styling information to guide an automatic selection and/or placement of the 3D model data within the 3D virtual environment.

10. The computer-implemented system of claim 1 , wherein user selection of products in the 2D image for placement in the 3D virtual environment comprises at least a first product and a second product forming a first grouping of products, wherein the rendering engine is configured to determine a starting element for the first grouping of products, where the starting element comprises the first product or the second product; and identifying a starting location for placement of the starting element wherein the starting element forms an anchor for rendering the first grouping of products in the 3D virtual environment and the rendering engine is further configured to render 3D models of the first grouping of products, with the 3D models of the products comprising the starting element being rendered at the starting location.

11. The computer-implemented system of claim 1 , further comprising:

a computer interface to a product inventory management system and the graphical user interface is configured to present a display option for a user to initiate purchase of one or more of the available product in the 2D image via the display option.

12. The computer-implemented system of claim 1 , wherein the resource database comprises a master resource database comprising a content management system configured to store product data, a set of data that keyed to a unique product identifier and 3D model data for the available product.

13. The computer-implemented system of claim 1 , wherein the first graphical user interface portion is further configured to provide a display option for user selection of a second 2D image portraying a room layout and the processor is further configured to automatically determine products, from the resource database, based on one or more functional rules associating products to one another.

14. The computer-implemented system of claim 1 , wherein the first graphical user interface portion is further configured to provide display information, comprising:

a first display portion for presenting an option for the user to select from the set of 2D images representing layouts of products in a room;

a second display portion for displaying objects from the 2D image;

a display option to enable the user to select an object from the displayed objects; and

third display portion for simultaneously displaying the 3D virtual environment; and

wherein the processor is further configured to:

identify from the 3D model data, in response to receiving a user selection of an object from the set of 2D images, 3D model data for the available product corresponding to the selected object;

generate the 3D model of the selected object for display in the 3D virtual environment; and

render the 3D model in the 3D virtual environment.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 13, 2024
From: BESECKER, BRET; BESECKER, BARRY; COWGILL, JEFFREY L., JR.; JEKELI, JONATHAN
To: MARXENT LABS LLC
Reel/Frame 066452/0810 →
Continuity (5)
Continuation 18061683 · Dec 5, 2022
Continuation 17360516 · Jun 28, 2021
Continuation 16391075 · Apr 22, 2019
Provisional Application 62660594 · Apr 20, 2018
Related Publication 20240212275A1 · Jun 27, 2024
References Cited (38)
US D669906S · Cranfill · 2012 [cited by applicant]
US D687047S · Hales · 2013 [cited by applicant]
US D716318S · Fan · 2014 [cited by applicant]
US 10572988B1 · Chaturvedi · 2020 [cited by applicant]
US 10600255B2 · Besecker · 2020 [cited by applicant]
US 10614625B1 · Goetzinger, Jr. · 2020 [cited by applicant]
US 10672191B1 · Besecker · 2020 [cited by applicant]
US 11049317B2 · Besecker · 2021 [cited by applicant]
US 11144680B2 · Jovanovic · 2021 [cited by applicant]
US 11783385B1 · Khosravan · 2023 [cited by applicant]
US 20020010655A1 · Jan · 2002 [cited by applicant]
US 20040183926A1 · Fukuda · 2004 [cited by applicant]
US 20050081161A1 · Macinnes · 2005 [cited by applicant]
US 20080122840A1 · Falco · 2008 [cited by applicant]
US 20130127893A1 · Gokturk · 2013 [cited by applicant]
US 20150243071A1 · Jovanovic · 2015 [cited by applicant]
US 20160026724A1 · Ramsey · 2016 [cited by applicant]
US 20190197599A1 · Zia · 2019 [cited by applicant]
US 20190325498A1 · Clark · 2019 [cited by applicant]
US 20190325643A1 · Besecker · 2019 [cited by applicant]
US 20200302681A1 · Totty · 2020 [cited by applicant]
US 20210049829A1 · Yang · 2021 [cited by applicant]
US 20210133850A1 · Ayush · 2021 [cited by applicant]
US 20210150805A1 · Stekovic · 2021 [cited by applicant]
US 20210279957A1 · Eder · 2021 [cited by applicant]
US 20220114291A1 · Li · 2022 [cited by applicant]
US 20220130126A1 · Delgado · 2022 [cited by applicant]
US 20230049221A1 · Rout · 2023 [cited by applicant]
US 20230173395A1 · Cella · 2023 [cited by applicant]
US 20230410436A1 · Beauchamp · 2023 [cited by applicant]
Cant, R. J., et al., “Methods for Automated Object Placement in Virtual Scenes”, 2009 11th International Conference on Computer Modelling and Simulation, Cambridge, UK, doi: 10.1109/UKSIM.2009.69., 2009 (Year: 2009), pp… [cited by applicant]
He, Yu, et al., “Context-Consistent Generation of Indoor Virtual Environments Based on Geometry Constraints,” IEEE Transactions on Visualization and Computer Graphics, vol. 28, No. 12, Dec. 2022 (Year: 2022), doi: 10.11… [cited by applicant]
Intemational Patent Application No. PCT/US2019/028574, Notification Concerning Transmittal of International Preliminary Report on Patentability (Chapter I of the Patent Cooperation Treaty) and International Preliminary … [cited by applicant]
Intemational Patent Application No. PCT/US2019/028574, Notification of Transmittal of the International Search Report and the Written Opinion of the International Searching Authority, or the Declaration, dated Jul. 24, … [cited by applicant]
Liu, Zicheng, et al., “Model-Driven Indoor Scenes Modeling from a Single Image,” Proceedings of the 41st Graphics Interface Conference, Jun. 3, 2015, 8 pages. [cited by applicant]
Satkin, Scott, et al., “Data-Driven Scene Understanding from 3D Models,” https://www.robots.ox.ac.uk/˜vgg/rg/pa/pers/bmvc2012_satkin_data_driven.pdf, dated Jan. 2012, retrieved Dec. 16, 2015, 11 pages. [cited by applicant]
Sweet Home, “Sweet Home 3D User's Guide;”, online at <URL: https://web.archive.org/web/20180208100505/http://www.sweethome3d.com:80/userGuide.jsp>, Feb. 8, 2018 (Year: 2018), pp. 1-15. [cited by applicant]
Xu, Kun, et al., “Sketch2Scene: Sketch-Based Co-Retrieval and Co-Placement of 3D Models,” ACM Transactions on Graphics, vol. 32, No. 4, Jul. 1, 2013, 12 pages. [cited by applicant]