IP Library Granted Patent US 12,277,642
Granted Patent B2
US 12,277,642 · App. 17/969,619 · Granted Apr 15, 2025

Localization failure handling on artificial reality systems

Inventors: Gioacchino Noris (Zurich, CH); Hongbing Hu (Palo Alto, CA); Anush Mohan (San Jose, CA)
G06T15/06G06T15/08G06T17/00G06T19/006G06T19/20G06T2200/24G06T2210/12G06T2219/004G06T2219/2004
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,277,642
App. No.
17/969,619
Granted
Apr 15, 2025
Kind
B2
Abstract

In particular embodiments, a computing system may initiate a scene alignment process to align a previous map of a scene with a current map of the scene. The system may send instructions to a user wearing an artificial-reality system to select a set of entities in the scene. The system may receive a selection of the set of entities in the scene. The system may determine a particular point in the scene based on an intersection of selected set of entities. The system may align the previous map with the current map based on the particular point in the scene. The system may load a scene model associated with the previous map into the current map.

Claims (76)

1. A method comprising, by a computing system:

initiating a scene alignment process to align a previous map of a scene with a current map of the scene;

sending instructions to a user wearing an artificial-reality system to select a set of entities in the scene;

receiving a selection of the set of entities in the scene;

determining a particular point in the scene based on an intersection of selected set of entities;

aligning the previous map with the current map based on the particular point in the scene; and

loading a scene model associated with the previous map into the current map.

2. The method of claim 1 , further comprising:

receiving a user selection of an application on the artificial-reality system, wherein the application requires a scene description associated with the scene model of the scene;

determining that the scene description is not associated with the current map;

determining that the previous map of the scene is present on the artificial-reality system; and

in response to determining that the scene description is not associated with the current map and that the previous map of the scene is present, initiating the scene alignment process to align the previous map of the scene with the current map of the scene.

3. The method of claim 1 , further comprising:

deleting the scene model from the previous map in response to loading the scene model into the current map.

4. The method of claim 1 , further comprising:

determining a plurality of maps of the scene present on the artificial-reality system;

instructing the user to select a particular map of the scene from the plurality of maps, wherein the particular map is the previous map; and

in response to the user selecting the particular map, initiating the scene alignment process to align the previous map of the scene with the current map of the scene.

5. The method of claim 1 , further comprising:

determining that the previous map of the scene is not present; and

in response to determining that the previous map of the scene is not present, initiating a scene capture process to generate a scene model of the scene,

wherein generated scene model is loaded into the current map.

6. The method of claim 1 , wherein aligning the previous map with the current map comprises:

re-positioning the previous map to align the particular point located in the previous map with the particular point located in the current map.

7. The method of claim 6 , wherein the scene model associated with the previous map is loaded into the current map in response to successful alignment of the particular point located in the previous map with the particular point located in the current map.

8. The method of claim 1 , wherein the scene is a living room of the user.

9. The method of claim 8 , wherein the set of entities in the scene comprises two walls of the living room.

10. The method of claim 9 , wherein the particular point is a room wall corner between the two walls.

11. One or more computer-readable non-transitory storage media embodying software that is operable, when executed by one or more processors, to:

initiate a scene alignment process to align a previous map of a scene with a current map of the scene;

send instructions to a user wearing an artificial-reality system to select a set of entities in the scene;

receive a selection of the set of entities in the scene;

determine a particular point in the scene based on an intersection of selected set of entities;

align the previous map with the current map based on the particular point in the scene; and

load a scene model associated with the previous map into the current map.

12. The one or more computer-readable non-transitory storage media of claim 11 , wherein the software is further operable when executed to:

receive a user selection of an application on the artificial-reality system, wherein the application requires a scene description associated with the scene model of the scene;

determine that the scene description is not associated with the current map;

determine that the previous map of the scene is present on the artificial-reality system; and

in response to determining that the scene description is not associated with the current map and that the previous map of the scene is present, initiate the scene alignment process to align the previous map of the scene with the current map of the scene.

13. The one or more computer-readable non-transitory storage media of claim 11 , wherein the software is further operable when executed to:

delete the scene model from the previous map in response to loading the scene model into the current map.

14. The one or more computer-readable non-transitory storage media of claim 11 , wherein the software is further operable when executed to:

determine a plurality of maps of the scene present on the artificial-reality system;

instruct the user to select a particular map of the scene from the plurality of maps, wherein the particular map is the previous map; and

in response to the user selecting the particular map, initiate the scene alignment process to align the previous map of the scene with the current map of the scene.

15. The one or more computer-readable non-transitory storage media of claim 11 , wherein the software is further operable when executed to:

determine that the previous map of the scene is not present; and

in response to determining that the previous map of the scene is not present,

initiate a scene capture process to generate a scene model of the scene,

wherein generated scene model is loaded into the current map.

16. A system comprising:

one or more processors; and

one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:

initiate a scene alignment process to align a previous map of a scene with a current map of the scene;

send instructions to a user wearing an artificial-reality system to select a set of entities in the scene;

receive a selection of the set of entities in the scene;

determine a particular point in the scene based on an intersection of selected set of entities;

align the previous map with the current map based on the particular point in the scene; and

load a scene model associated with the previous map into the current map.

17. The system of claim 16 , wherein the one or more processors are further operable when executing the instructions to cause the system to:

receive a user selection of an application on the artificial-reality system, wherein the application requires a scene description associated with the scene model of the scene;

determine that the scene description is not associated with the current map;

determine that the previous map of the scene is present on the artificial-reality system; and

in response to determining that the scene description is not associated with the current map and that the previous map of the scene is present, initiate the scene alignment process to align the previous map of the scene with the current map of the scene.

18. The system of claim 16 , wherein the one or more processors are further operable when executing the instructions to cause the system to:

delete the scene model from the previous map in response to loading the scene model into the current map.

19. The system of claim 16 , wherein the one or more processors are further operable when executing the instructions to cause the system to:

determine a plurality of maps of the scene present on the artificial-reality system;

instruct the user to select a particular map of the scene from the plurality of maps, wherein the particular map is the previous map; and

in response to the user selecting the particular map, initiate the scene alignment process to align the previous map of the scene with the current map of the scene.

20. The system of claim 16 , wherein the one or more processors are further operable when executing the instructions to cause the system to:

determine that the previous map of the scene is not present; and

in response to determining that the previous map of the scene is not present,

initiate a scene capture process to generate a scene model of the scene,

wherein generated scene model is loaded into the current map.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 15, 2023
From: NORIS, GIOACCHINO; HU, HONGBING; MOHAN, ANUSH
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 064591/0372 →
Continuity (2)
Provisional Application 63272092 · Oct 26, 2021
Related Publication 20230125390A1 · Apr 27, 2023
References Cited (65)
US 9063330B2 · LaValle et al. · 2015 [cited by applicant]
US 9964409B1 · Flint et al. · 2018 [cited by applicant]
US 10338392B2 · Kohler et al. · 2019 [cited by applicant]
US 10346623B1 · Brandwine et al. · 2019 [cited by applicant]
US 10466953B2 · Eade et al. · 2019 [cited by applicant]
US 10657701B2 · Osman et al. · 2020 [cited by applicant]
US 11024079B1 · Chuah · 2021 [cited by examiner]
US 11158130B1 · Rubaiat Habib et al. · 2021 [cited by applicant]
US 11222468B1 · Lovegrove et al. · 2022 [cited by applicant]
US 11295460B1 · Aghdasi · 2022 [cited by examiner]
US 11481925B1 · Li et al. · 2022 [cited by applicant]
US 11670045B2 · Zhang et al. · 2023 [cited by applicant]
US 20050249426A1 · Badawy · 2005 [cited by applicant]
US 20120300020A1 · Arth et al. · 2012 [cited by applicant]
US 20120306850A1 · Balan et al. · 2012 [cited by applicant]
US 20140267234A1 · Hook et al. · 2014 [cited by applicant]
US 20150029218A1 · Williams et al. · 2015 [cited by applicant]
US 20150062125A1 · Aguilera Perez et al. · 2015 [cited by applicant]
US 20150331970A1 · Jovanovic · 2015 [cited by applicant]
US 20160364912A1 · Cho et al. · 2016 [cited by applicant]
US 20170115488A1 · Ambrus et al. · 2017 [cited by applicant]
US 20170243403A1 · Daniels et al. · 2017 [cited by applicant]
US 20170337749A1 · Nerurkar et al. · 2017 [cited by applicant]
US 20170345167A1 · Ard et al. · 2017 [cited by applicant]
US 20180053329A1 · Roberts et al. · 2018 [cited by applicant]
US 20180122139A1 · Janzer et al. · 2018 [cited by applicant]
US 20180143023A1 · Bjorke et al. · 2018 [cited by applicant]
US 20180143756A1 · Mildrew et al. · 2018 [cited by applicant]
US 20180144547A1 · Shakib et al. · 2018 [cited by applicant]
US 20180232937A1 · Moyer et al. · 2018 [cited by applicant]
US 20190026956A1 · Gausebeck et al. · 2019 [cited by applicant]
US 20190051054A1 · Jovanovic et al. · 2019 [cited by applicant]
US 20190236842A1 · Bennett et al. · 2019 [cited by applicant]
US 20190287311A1 · Bhatnagar et al. · 2019 [cited by applicant]
US 20200066046A1 · Stahl et al. · 2020 [cited by applicant]
US 20200099954A1 · Hemmer et al. · 2020 [cited by applicant]
US 20200175764A1 · Romea et al. · 2020 [cited by applicant]
US 20200250879A1 · Foster et al. · 2020 [cited by applicant]
US 20200302681A1 · Totty et al. · 2020 [cited by applicant]
US 20200364901A1 · Choudhuri et al. · 2020 [cited by applicant]
US 20210056762A1 · Robbe et al. · 2021 [cited by applicant]
US 20210304509A1 · Berkebile · 2021 [cited by applicant]
US 20210326026A1 · Osipov et al. · 2021 [cited by applicant]
US 20220043446A1 · Ding et al. · 2022 [cited by applicant]
US 20220122285A1 · Suleiman et al. · 2022 [cited by applicant]
US 20220130064A1 · Tomar · 2022 [cited by examiner]
US 20220254207A1 · Billy et al. · 2022 [cited by applicant]
US 20220269885A1 · Wixson et al. · 2022 [cited by applicant]
US 20230127307A1 · Noris et al. · 2023 [cited by applicant]
US 20230237692A1 · Alaghi et al. · 2023 [cited by applicant]
EP 3419286A1 · 2018 [cited by applicant]
WO 2013155217A1 · 2013 [cited by applicant]
WO 2015192117A1 · 2015 [cited by applicant]
WO 2021010660A1 · 2021 [cited by applicant]
WO 2021188741A1 · 2021 [cited by applicant]
Balntas V., et al., “HPatches: A Benchmark and Evaluation of Handcrafted and Learned Local Descriptors,” Computer Vision and Pattern Recognition (CVPR), Apr. 19, 2017, arXiv:1704.05939v1 [cs.CV], 10 Pages. [cited by applicant]
International Preliminary Report on Patentability for International Application No. PCT/US2022/047864, mailed May 10, 2024, 12 pages. [cited by applicant]
International Preliminary Report on Patentability for International Application No. PCT/US2023/011579, mailed Aug. 8, 2024, 10 pages. [cited by applicant]
International Search Report and Written Opinion for International Application No. PCT/US2020/027763, mailed Jul. 23, 2020, 12 Pages. [cited by applicant]
International Search Report and Written Opinion for International Application No. PCT/US2022/052472, mailed Apr. 17, 2023, 11 pages. [cited by applicant]
International Search Report and Written Opinion for International Application No. PCT/US2023/011579, mailed May 17, 2023, 12 pages. [cited by applicant]
Morrison J. G., et al., “Scalable Multirobot Localization and Mapping with Relative Maps: Introducing Moarslam,” IEEE Control Systems, vol. 36, No. 2, Apr. 1, 2016, pp. 75-85. [cited by applicant]
Mur-Artal R., et al., “ORB-SLAM: a Versatile and Accurate Monocular SLAM System,” IEEE Transactions on Robotics, Sep. 18, 2015, arXiv:1502.00956v2 [cs.RO], 18 Pages, DOI: 10.1109/TRO.2015.2463671. [cited by applicant]
Tian Y., et al., “SOSNet: Second Order Similarity Regularization for Local Descriptor Learning,” Computer Vision and Pattern Recognition (CVPR), Dec. 16, 2019, arXiv:1904.05019v2 [cs.CV], 10 Pages. [cited by applicant]
International Search Report and Written Opinion for International Application No. PCT/US2022/047864, mailed Apr. 6, 2023, 15 pages. [cited by applicant]