IP Library Granted Patent US 12,260,640
Granted Patent B2
US 12,260,640 · App. 18/596,078 · Granted Mar 25, 2025

Virtualizing objects using object models and object position data

Inventors: Mason E. Sheffield (Woodinville, WA); Josh Shabtai (Woodinville, WA)
Assignee: Lowe's Companies, Inc.
G06V20/20G06F16/5854G06F16/9537G06N20/00G06T7/75G06T19/006G06T2207/10016G06T2207/10028
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,260,640
App. No.
18/596,078
Granted
Mar 25, 2025
Kind
B2
Abstract

Described herein are techniques for receiving, from a first device, first data about a physical object located in a physical space. The techniques further including storing, in a catalog of objects generated for the physical space, information about the physical object and indicating at least a first location of the physical object in the physical space. The techniques include receiving, from a second device, second data about the physical space, the second data sent from the second device upon or after an occurrence of an event. The techniques include determining, based on the second data, at least an impact area of the event to the physical space and determining, based on the catalog of objects and the second data, whether the physical object is impacted by the event. The techniques include causing an output indicating whether the physical object is impacted by the event.

Claims (47)

1. A method comprising:

receiving, from a first device, first data about a physical object located in a physical space;

storing, in a catalog of objects generated for the physical space, information about the physical object and indicating at least a first location of the physical object in the physical space;

receiving, from a second device, second data about the physical space, the second data sent from the second device upon or after an occurrence of an event, the second device being the same as or different from the first device;

determining, based on the second data, at least an impact area of the event to the physical space;

determining, based on the catalog of objects and the second data, whether the physical object is impacted by the event; and

causing an output indicating whether the physical object is impacted by the event.

2. The method of claim 1 , wherein receiving the first data includes receiving sensor data generated by one or more sensors of the first device.

3. The method of claim 1 , further comprising generating the information by at least:

determining, based on the first data, the first location of the physical object;

mapping the physical object to one or more corresponding physical object models within an object model database;

identifying that the physical object is to be associated with the first location; and

including, in the information, an indication of the physical object model and the first location of the physical object.

4. The method of claim 1 , wherein storing the information in the catalog of objects comprises storing a representation of the physical object based on the first data.

5. The method of claim 4 , wherein the first data includes image data, and wherein the representation is generated based on the image data.

6. The method of claim 1 , wherein the first data includes depth information associated with the physical object.

7. The method of claim 6 , wherein the information about the physical object includes a first position of the physical object determined using the depth information associated with the physical object.

8. The method of claim 7 , wherein the first location comprises the first position and a first orientation.

9. The method of claim 1 , wherein the second data about the physical space comprises image data.

10. The method of claim 9 , wherein the impact area is determined based on user input indicating a boundary of the impact area in at least one image.

11. The method of claim 9 , wherein the image data comprises at least one of color data or texture data corresponding to the physical space or the physical object and the impact area is determined by using at least one of the color data or the texture data.

12. A system comprising:

one or more processors; and

one or more memory storing instructions that upon execution by the one or more processors, configure the system to:

receive, from a first device, first data about a physical object located in a physical space;

store, in a catalog of objects generated for the physical space, information about the physical object and indicating at least a first location of the physical object in the physical space;

receive, from a second device, second data about the physical space, the second data sent from the second device upon or after an occurrence of an event, the second device being the same as or different from the first device;

determine, based on the second data, at least an impact area of the event to the physical space;

determine, based on the catalog of objects and the second data, whether the physical object is impacted by the event; and

cause an output indicating whether the physical object is impacted by the event.

13. The system of claim 12 , wherein the second data corresponds to a floorplan layout of the physical space.

14. The system of claim 13 , wherein the instructions, upon execution by the one or more processors, further configure the system to determine the impact area based on user input indicating a boundary of the impact area and on the floorplan layout.

15. The system of claim 12 , wherein the instructions, upon execution by the one or more processors, further configure the system to associate the first location with the impact area if the first location is determined to be within the impact area.

16. The system of claim 12 , wherein the instructions, upon execution by the one or more processors, further configure the system to determine that the physical object is impacted by the event if the first location is determined to be within the impact area.

17. The system of claim 12 , wherein the instructions, upon execution by the one or more processors, further configure the system to:

determine, based on the second data, a second location of the physical object, wherein the physical object is determined to be impacted by the event if a determination is made that the first location does not correspond to the second location of the physical object.

18. One or more non-transitory computer-readable storage media storing instructions that upon execution on a system, cause the system to perform operations comprising:

receiving, from a first device, first data about a physical object located in a physical space;

storing, in a catalog of objects generated for the physical space, information about the physical object and indicating at least a first location of the physical object in the physical space;

receiving, from a second device, second data about the physical space, the second data sent from the second device upon or after an occurrence of an event, the second device being the same as or different from the first device;

determining, based on the second data, at least an impact area of the event to the physical space;

determining, based on the catalog of objects and the second data, whether the physical object is impacted by the event; and

causing an output indicating whether the physical object is impacted by the event.

19. The one or more non-transitory computer-readable storage media of claim 18 , wherein the operations further comprise:

determining, based on the second data, a second location of the physical object, wherein the second data about the physical space includes image data, wherein the image data comprises at least one of color data or texture data corresponding to the physical object, and wherein the physical object is determined to be impacted by the event if a determination is made that at least one of the color data or the texture data does not correspond to a color property of the physical object or a texture property of the physical object stored in the catalog.

20. The one or more non-transitory computer-readable storage media of claim 18 , wherein the operations further comprise:

determining, based on the second data, a second location of the physical object, wherein causing the output includes identifying the physical object and an impact assessment for the physical object on a user interface of the second device.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 8, 2024
From: SHEFFIELD, MASON E.; SHABTAI, JOSH
To: LOWE'S COMPANIES INC.
Reel/Frame 066700/0746 →
Continuity (5)
Continuation 18186508 · Mar 20, 2023
Continuation 17343009 · Jun 9, 2021
Continuation 16222333 · Dec 17, 2018
Continuation 15840567 · Dec 13, 2017
Related Publication 20240212344A1 · Jun 27, 2024
References Cited (158)
US 7391424B2 · Lonsing · 2008 [cited by applicant]
US 7457730B2 · Degnan et al. · 2008 [cited by applicant]
US 7627502B2 · Cheng et al. · 2009 [cited by applicant]
US 7933068B2 · Peterson et al. · 2011 [cited by applicant]
US 8170367B2 · Moody et al. · 2012 [cited by applicant]
US 8463025B2 · Melvin et al. · 2013 [cited by applicant]
US 8668498B2 · Calman et al. · 2014 [cited by applicant]
US 8687104B2 · Penov et al. · 2014 [cited by applicant]
US 8797354B2 · Noge · 2014 [cited by applicant]
US 8866847B2 · Bedi et al. · 2014 [cited by applicant]
US 9001154B2 · Meier et al. · 2015 [cited by applicant]
US 9129404B1 · Wagner · 2015 [cited by applicant]
US 9164577B2 · Tapley et al. · 2015 [cited by applicant]
US 9373195B2 · Kasahara · 2016 [cited by applicant]
US 9418378B2 · Staicut et al. · 2016 [cited by applicant]
US 9449324B2 · Shintani et al. · 2016 [cited by applicant]
US 9552650B2 · Furuya · 2017 [cited by applicant]
US 9563825B2 · Shen et al. · 2017 [cited by applicant]
US 9697416B2 · Shen et al. · 2017 [cited by applicant]
US 9728010B2 · Thomas et al. · 2017 [cited by applicant]
US 9965800B1 · Pitstick et al. · 2018 [cited by applicant]
US 10163271B1 · Powers et al. · 2018 [cited by applicant]
US 10192115B1 · Sheffield et al. · 2019 [cited by applicant]
US 10210564B2 · Qi et al. · 2019 [cited by applicant]
US 10395435B2 · Powers et al. · 2019 [cited by applicant]
US 11062139B2 · Sheffield et al. · 2021 [cited by applicant]
US 11062383B2 · Holt et al. · 2021 [cited by applicant]
US 11348327B1 · Ha et al. · 2022 [cited by applicant]
US 11521273B2 · Jakka et al. · 2022 [cited by applicant]
US 11599950B2 · Spader et al. · 2023 [cited by applicant]
US 11615597B2 · Rockel et al. · 2023 [cited by applicant]
US 11615619B2 · Sheffield et al. · 2023 [cited by applicant]
US 11640486B2 · Santarone et al. · 2023 [cited by applicant]
US 11836216B1 · McIntyre · 2023 [cited by examiner]
US 11875396B2 · Holt et al. · 2024 [cited by applicant]
US 12087054B2 · Sheffield · 2024 [cited by examiner]
US 20010034668A1 · Whitworth · 2001 [cited by applicant]
US 20040201823A1 · Raskar et al. · 2004 [cited by applicant]
US 20050035980A1 · Lonsing · 2005 [cited by applicant]
US 20060137260A1 · Shernaman · 2006 [cited by applicant]
US 20080071559A1 · Arrasvuori · 2008 [cited by applicant]
US 20080162262A1 · Perkins · 2008 [cited by applicant]
US 20080191864A1 · Wolfson · 2008 [cited by applicant]
US 20090190858A1 · Moody et al. · 2009 [cited by applicant]
US 20100321382A1 · Amaratunga et al. · 2010 [cited by applicant]
US 20110202396A1 · Viveiros et al. · 2011 [cited by applicant]
US 20120062596A1 · Bedi et al. · 2012 [cited by applicant]
US 20120127199A1 · Aarabi · 2012 [cited by applicant]
US 20120231424A1 · Calman et al. · 2012 [cited by applicant]
US 20130106910A1 · Sacco · 2013 [cited by applicant]
US 20130106920A1 · Park et al. · 2013 [cited by applicant]
US 20130196772A1 · Latta et al. · 2013 [cited by applicant]
US 20130215116A1 · Siddique et al. · 2013 [cited by applicant]
US 20130249948A1 · Reitan · 2013 [cited by applicant]
US 20130305638A1 · Wheeler · 2013 [cited by applicant]
US 20140072213A1 · Paiton et al. · 2014 [cited by applicant]
US 20140082610A1 · Wang et al. · 2014 [cited by applicant]
US 20140160115A1 · Keitler et al. · 2014 [cited by applicant]
US 20140171039A1 · Bjontegard · 2014 [cited by applicant]
US 20140176565A1 · Adeyoola et al. · 2014 [cited by applicant]
US 20140279242A1 · Staicut et al. · 2014 [cited by applicant]
US 20140285522A1 · Kim et al. · 2014 [cited by applicant]
US 20150179147A1 · Rezaiifar et al. · 2015 [cited by applicant]
US 20150206292A1 · Masuko et al. · 2015 [cited by applicant]
US 20150348313A1 · Fuchikami et al. · 2015 [cited by applicant]
US 20160048497A1 · Goswami · 2016 [cited by applicant]
US 20160051066A1 · Koch · 2016 [cited by applicant]
US 20160086078A1 · Ji et al. · 2016 [cited by applicant]
US 20160174902A1 · Georgescu et al. · 2016 [cited by applicant]
US 20160180193A1 · Masters et al. · 2016 [cited by applicant]
US 20160189426A1 · Thomas et al. · 2016 [cited by applicant]
US 20160210602A1 · Siddique et al. · 2016 [cited by applicant]
US 20160217615A1 · Kraver · 2016 [cited by applicant]
US 20160275376A1 · Kant · 2016 [cited by applicant]
US 20160300293A1 · Nagar · 2016 [cited by applicant]
US 20160364793A1 · Sacco · 2016 [cited by applicant]
US 20160371884A1 · Benko et al. · 2016 [cited by applicant]
US 20170011558A1 · Meier et al. · 2017 [cited by applicant]
US 20170021273A1 · Rios · 2017 [cited by applicant]
US 20170039613A1 · Kaehler et al. · 2017 [cited by applicant]
US 20170039986A1 · Lanier et al. · 2017 [cited by applicant]
US 20170123750A1 · Todasco · 2017 [cited by applicant]
US 20170132497A1 · Santos et al. · 2017 [cited by applicant]
US 20170140236A1 · Price et al. · 2017 [cited by applicant]
US 20170154425A1 · Pierce et al. · 2017 [cited by applicant]
US 20170161592A1 · Su et al. · 2017 [cited by applicant]
US 20170173466A1 · Fahmie · 2017 [cited by applicant]
US 20170178272A1 · Lashkari et al. · 2017 [cited by applicant]
US 20170200313A1 · Lee et al. · 2017 [cited by applicant]
US 20170229154A1 · Bose et al. · 2017 [cited by applicant]
US 20170330273A1 · Holt et al. · 2017 [cited by applicant]
US 20180060946A1 · Devries · 2018 [cited by applicant]
US 20180068489A1 · Kim et al. · 2018 [cited by applicant]
US 20180350144A1 · Rathod · 2018 [cited by applicant]
US 20190172261A1 · Alt et al. · 2019 [cited by applicant]
US 20190180104A1 · Sheffield et al. · 2019 [cited by applicant]
US 20190318404A1 · LaMontagne · 2019 [cited by applicant]
US 20200327733A1 · Mueller · 2020 [cited by examiner]
US 20210158624A1 · Moon et al. · 2021 [cited by applicant]
US 20210209862A1 · Seva et al. · 2021 [cited by applicant]
US 20210295052A1 · Sheffield et al. · 2021 [cited by applicant]
US 20220129974A1 · Delgado et al. · 2022 [cited by applicant]
US 20220292543A1 · Henderson · 2022 [cited by examiner]
US 20230169598A1 · Pedersen · 2023 [cited by examiner]
KR 20130137968A · 2013 [cited by applicant]
KR 20140081729A · 2014 [cited by applicant]
KR 20140082610A · 2014 [cited by applicant]
WO 2015192117A1 · 2015 [cited by applicant]
WO 2016075081A1 · 2016 [cited by applicant]
WO 2017100658A1 · 2017 [cited by applicant]
WO 2017206451A1 · 2017 [cited by applicant]
WO 2019118599A3 · 2019 [cited by applicant]
U.S. Appl. No. 18/186,508, “Notice of Allowance”, Apr. 9, 2024, 12 pages. [cited by applicant]
“3D Room Designer”, Living Spaces, livingspaces.com, Jul. 2017, 5 pages. [cited by applicant]
“Display Mapper”, Displaymapping4, Available Online at, web.archive.org/web/20160407215821/http://www.displaymapper.com:80/, Accessed from Internet on: Jul. 12, 2021, 6 pages. [cited by applicant]
U.S. Appl. No. 15/150,973 , “Final Office Action”, Apr. 10, 2019, 15 pages. [cited by applicant]
U.S. Appl. No. 15/150,973 , “Final Office Action”, Nov. 5, 2019, 17 pages. [cited by applicant]
U.S. Appl. No. 15/150,973 , “Non-Final Office Action”, Dec. 21, 2018, 12 pages. [cited by applicant]
U.S. Appl. No. 15/150,973 , “Non-Final Office Action”, Jul. 25, 2019, 18 pages. [cited by applicant]
U.S. Appl. No. 15/150,973 , “Notice of Allowance”, Mar. 10, 2021, 18 pages. [cited by applicant]
U.S. Appl. No. 15/840,567 , “Non-Final Office Action”, Feb. 8, 2018, 35 pages. [cited by applicant]
U.S. Appl. No. 15/840,567 , “Notice of Allowance”, Sep. 17, 2018, 12 pages. [cited by applicant]
U.S. Appl. No. 16/222,333 , “Final Office Action”, Jan. 15, 2021, 27 pages. [cited by applicant]
U.S. Appl. No. 16/222,333 , “Non-Final Office Action”, Jul. 14, 2020, 26 pages. [cited by applicant]
U.S. Appl. No. 16/222,333 , “Notice of Allowance”, Mar. 19, 2021, 12 pages. [cited by applicant]
U.S. Appl. No. 17/343,009 , “Non-Final Office Action”, Aug. 16, 2022, 10 pages. [cited by applicant]
U.S. Appl. No. 17/343,009 , “Notice of Allowance”, Oct. 13, 2022, 11 pages. [cited by applicant]
U.S. Appl. No. 17/367,122 , “Non-Final Office Action”, May 25, 2023, 33 pages. [cited by applicant]
U.S. Appl. No. 17/367,122 , “Notice of Allowance”, Sep. 12, 2023, 7 pages. [cited by applicant]
U.S. Appl. No. 18/186,508 , “Notice of Allowance”, Nov. 24, 2023, 11 pages. [cited by applicant]
CA2,966,447 , “Notice of Allowance”, Sep. 4, 2019, 1 page. [cited by applicant]
CA2,966,447 , “Office Action”, Oct. 4, 2018, 4 pages. [cited by applicant]
CA3,085,428 , “Notice of Allowance”, Feb. 10, 2023, 1 page. [cited by applicant]
CA3,085,428 , “Office Action”, Nov. 24, 2022, 4 pages. [cited by applicant]
Cebulla , “Projection-Based Augmented Reality”, Distributed Systems Seminar, 2013, 10 pages. [cited by applicant]
Cheng et al., “Locality-Sensitive Deconvolution Networks with Gated Fusion for RGB-D Indoor Semantic Segmentation”, Computer Vision Foundation, 2017, pp. 3029-3037. [cited by applicant]
Hein , “3D Video Projection Mapping”, Displaymapping 1, Image Audiovisuals, Jun. 12, 2015. [cited by applicant]
Izadinia et al., “IM2CAD”, arXiv preprint arXiv:1608.05137, 2016, 11 pages. [cited by applicant]
Jatnes , “The Future of Retail: Projection-Mapped Mannequins & Interactive Window Displays”, Displaymapping3, The Visualist, Oct. 4, 2012. [cited by applicant]
Kang et al., “Discovering Object Instances From Scenes of Daily Living”, Computer Vision (ICCV), IEEE International Conference, 2011, 9 pages. [cited by applicant]
Li et al., “Rgb-d Scene Labeling With Long Short-term Memorized Fusion Model”, arXiv preprint arXiv:1604.05000, 2016, 17 pages. [cited by applicant]
Marner et al., “Spatial User Interfaces for Large-Scale Projector-Based Augmented Reality”, IEEE Computer Graphics and Applications, vol. 34, Issue 6, Nov.-Dec. 2014, pp. 74-82. [cited by applicant]
Massa et al., “Deep Exemplar 2d-3d Detection by Adapting From Real to Rendered Views”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 6024-6033. [cited by applicant]
McCormac et al., “Semanticfusion: Dense 3d Semantic Mapping With Convolutional Neural Networks”, Robotics and Automation (ICRA), IEEE International Conference, 2017, 7 pages. [cited by applicant]
Neale , “Transforming Retail Display Through Scalable and Affordable Projection Mapping”, Displaymapping2, DailyDOOH, Feb. 19, 2016. [cited by applicant]
PCT/US2018/065208 , “International Preliminary Report on Patentability”, Jun. 25, 2020, 9 pages. [cited by applicant]
PCT/US2018/065208 , “International Search Report and Written Opinion”, Jul. 2, 2019, 12 pages. [cited by applicant]
Qi et al., “PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation”, Institute of Electrical and Electronics Engineer, Conference on Computer Vision and Pattern Recognition (CVPR), Apr. 10, 2017, p… [cited by applicant]
Schenker , “Map Projections Like the Best of 'em With This Madmapper Tutorial”, Vice News, Aug. 12, 2011. [cited by applicant]
Thomas et al., “Spatial Augmented Reality—a Tool for 3D Data Visualization”, IEEE VIS International Workshop on 3DVis (3DVis), Nov. 9, 2014. [cited by applicant]
Unal et al., “Distant Augmented Reality: Bringing a New Dimension to User Experience Using Drones”, Digital Applications in Archaeology and Cultural Heritage, vol. 17, 2020, pp. 1-12. [cited by applicant]
Wu et al., “Single Image 3D Interpreter Network”, European Conference on Computer Vision, Springer International Publishing, Oct. 4, 2016, pp. 1-18. [cited by applicant]
Xiang et al., “ObjectNet3D: A Large Scale Database for 3D Object Recognition”, Available Online at: https://cvgl.stanford.edu/papers/xiang_eccv16.pdf, Oct. 2016, 16 pages. [cited by applicant]
Yurieff , “This Shopping App Lets You See a Virtual Couch in Your Real Living Room”, Available Online at: https://money.cnn.com/2017/05/03/technology/houzz-3d-augmented-reality-shopping/index.html, May 3, 2017, 5 pages. [cited by applicant]
Zahavy et al., “Is a Picture Worth a Thousand Words? A Deep Multi-Modal Fusion Architecture for Product Classification in E-Commerce”, Available Online at: https://arxiv.org/pdf/1611.09534.pdf, Nov. 29, 2016, pp. 1-10. [cited by applicant]
Zhao et al., “Image Parsing via Stochastic Scene Grammar”, NIPS'11: Proceedings of the 24th International Conference on Neural Information Processing Systems, Dec. 2011, pp. 1-9. [cited by applicant]
Zhao et al., “Scene Parsing by Integrating Function, Geometry and Appearance Models”, IEEE Conference on Computer Vision and Pattern Recognition, Jun. 23-28, 2013, pp. 3119-3126. [cited by applicant]
CA3198605 , “Office Action”, Sep. 12, 2024, 4 pages. [cited by applicant]