IP Library › Granted Patent US 12,724,822
Granted Patent B2
US 12,724,822 · App. 18/660,610 · Granted Sep 1, 2026

Montaging system

Inventors: Hector H. Gonzalez-Banos (San Mateo, CA); Ramya Narasimha (San Francisco, CA); Max McFarland (Lake Oswego, OR)
Assignee: Insightful Mechanisms LLC
G06F16/787G06T7/20G06T7/73H04N7/188H04N23/66G06T2207/10016
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,724,822
App. No.
18/660,610
Granted
Sep 1, 2026
Kind
B2
Abstract

Montaging techniques are disclosed that utilize non-sequential visual inertial odometry (VIO) performed on non-sequential/unordered capture data collected by a capture apparatus. The capture apparatus is carried by an observer/inspector at a site during a walkthrough/inspection. The capture apparatus comprises one or more cameras and an inertial measurement unit (IMU). User markings are applied to portions of the capture data. Based on the non-sequential VIO, a velocity profile and subsequently a set of positions of the capture apparatus are estimated as it was carried by the observer/inspector during the walkthrough/inspection. The above is accomplished via a constrained integration that utilizes constraints conditioning the motion of the capture apparatus. A montage of the capture data is produced that suits the needs of a given application of the instant montaging technology.

Claims (49)

1 . A montaging system comprising:

(a) a capture apparatus containing a camera and an inertial measurement unit (IMU);

(b) computer-readable instructions stored in a non-transitory storage medium and at least one microprocessor coupled to said non-transitory storage medium for executing said computer-readable instructions, said at least one microprocessor configured to:

collect one or more portions of capture data produced by said capture apparatus carried by a user undergoing motion at a site during a capture session;

allow said user to apply one or more markings to said one or more portions;

apply one or more constraints conditioning said motion;

perform an estimation of a velocity profile of said capture apparatus from said one or more portions by employing non-sequential visual inertial odometry (VIO) and by utilizing said one or more markings;

determine a plurality of positions of said capture apparatus by performing a constrained integration of said velocity profile based on said one or more constraints; and

produce a montage of said capture data based on said plurality of positions.

2 . The montaging system of claim 1 , wherein said at least one microprocessor is further configured to store said one or more portions locally on said capture apparatus and then upload them to a remote storage location.

3 . The montaging system of claim 2 , wherein a portion amongst said one or more portions is skipped from uploading to said remote storage location based on said one or more markings.

4 . The montaging system of claim 1 , wherein said one or more markings comprise a waypoint entered by said user.

5 . The montaging system of claim 1 , wherein said one or more constraints are based on one or more elements of a group including said one or more markings, a correction entered by said user, a reference point derived from a fiducial marker at said site, a pause detected in said motion, and a known compass point at said site.

6 . The montaging system of claim 1 , wherein said one or more portions are first ordered in accordance with said montage before said estimation.

7 . The montaging system of claim 1 , wherein said camera is a 360-degree camera and wherein said montage is a 360-degree virtual tour.

8 . The montaging system of claim 1 , wherein said montage is a hyperlapse.

9 . The montaging system of claim 1 , wherein said capture apparatus is head-mounted onto said user during said capture session.

10 . The montaging system of claim 1 , further comprising a companion device for issuing commands to said capture apparatus, said companion device carried by said user during said capture session.

11 . The montaging system of claim 10 , wherein said companion device is one of a smartwatch, a smartphone, a wearable device, a tablet, a laptop, a personal digital assistant (PDA), and a mobile computing device.

12 . The montaging system of claim 1 , wherein said camera is in an array of cameras contained in said capture apparatus, said array of cameras producing a 360-degree video footage contained in said capture data.

13 . The montaging system of claim 1 , wherein said at least one microprocessor is further configured to enable said user to include one or both of pictures and notes in said capture data, said pictures and notes taken using a secondary device carried by said user during said capture session.

14 . The montaging system of claim 1 , wherein said capture apparatus is an on-off device (OOD) and wherein said one or more markings comprise a start and an end entered by said user concurrently with the start and the end of said capture session respectively.

15 . The montaging system of claim 1 , wherein said capture apparatus is an always-on device (AOD) and wherein said one or more markings comprise a start and an end of said capture session entered retrospectively by said user after said capture session.

16 . The montaging system of claim 1 , wherein said capture session is an inspection performed by said user at said site and wherein said plurality of positions trace a path of said user traversed during said inspection.

17 . The montaging system of claim 16 , wherein said montage comprises said path fitted to a blueprint of said site.

18 . The montaging system of claim 16 , wherein said at least one microprocessor is further configured to enable said user to perform an assignment of said inspection to said site.

19 . A montaging system comprising:

(a) one or more capture apparatus each containing a camera and an inertial measurement unit (IMU);

(b) computer-readable instructions stored in non-transitory storage media and at least one microprocessor coupled to said non-transitory storage media for executing said computer-readable instructions, said at least one microprocessor configured to:

collect one or more portions of capture data produced by each of said one or more capture apparatus carried by one or more respective users each undergoing motion at a site during a capture session;

allow said one or more respective users to apply one or more markings to said one or more portions;

apply one or more constraints conditioning said motion;

perform an estimation of a velocity profile of each of said capture apparatus from said one or more portions by employing non-sequential visual inertial odometry (VIO) and by utilizing said one or more markings;

determine a plurality of positions of each of said capture apparatus by performing a constrained integration of said velocity profile based on said one or more constraints; and

produce a montage of said capture data based on said plurality of positions.

20 . A computer-implemented montaging method comprising the steps of:

(a) collecting one or more portions of capture data produced by a capture apparatus carried by a user undergoing motion at a site during a capture session, said capture apparatus comprising a camera and an inertial measurement unit (IMU);

(b) applying one or more markings by said user to said one or more portions;

(c) applying one or more constraints conditioning said motion;

(d) estimating a velocity profile of said capture apparatus from said one or more portions by employing non-sequential visual inertial odometry (VIO) and by utilizing said one or more markings;

(e) determining a plurality of positions of said capture apparatus by performing a constrained integration of said velocity profile based on said one or more constraints; and

(f) producing a montage of said capture data based on said plurality of positions.

21 . The computer-implemented montaging method of claim 20 , excluding a portion amongst said one or more portions from said montage based on said one or more markings.

22 . The computer-implemented montaging method of claim 20 , providing said one or more markings to comprise a waypoint marking.

23 . The computer-implemented montaging method of claim 20 , basing said one or more constraints on one or more elements of a group containing said one or more markings, a correction entered by said user, a reference point derived from a landmark at said site, a pause detected in said motion, and a known heading at said site.

24 . The computer-implemented montaging method of claim 20 , ordering said one or more portions according to said montage before said step (d).

25 . The computer-implemented montaging method of claim 20 , wherein said capture session is an inspection performed by said user at said site and wherein said plurality of positions trace a path of said user travelled during said inspection.

26 . The computer-implemented montaging method of claim 25 , fitting said path to a blueprint of said site for said montage.

27 . The computer-implemented montaging method of claim 26 , basing said fitting on a confidence measure deriving from said non-sequential VIO.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 10, 2024
From: GONZALEZ-BANOS, HECTOR H.; NARASIMHA, RAMYA; MCFARLAND, MAX
To: INSIGHTFUL MECHANISMS LLC
Reel/Frame 067377/0842 →
Continuity (1)
Related Publication 20250348540A1 · Nov 13, 2025
References Cited (87)
US 9318023B2 · Moshchuk et al. · 2016 [cited by applicant]
US 10571926B1 · Zhang et al. · 2020 [cited by applicant]
US 10907971B2 · Roumeliotis et al. · 2021 [cited by applicant]
US 10953545B2 · Hong et al. · 2021 [cited by applicant]
US 11030478B1 · Devitt et al. · 2021 [cited by applicant]
US 11188787B1 · Ulbricht et al. · 2021 [cited by applicant]
US 11380362B2 · Huang · 2022 [cited by applicant]
US 11399531B1 · Sibley et al. · 2022 [cited by applicant]
US 11496723B1 · Nepveu et al. · 2022 [cited by applicant]
US 11521332B1 · Chen et al. · 2022 [cited by applicant]
US 11544867B2 · Zhang et al. · 2023 [cited by applicant]
US 11573562B2 · Bachrach et al. · 2023 [cited by applicant]
US 20110282622A1 · Canter · 2011 [cited by applicant]
US 20120065871A1 · Deshpande et al. · 2012 [cited by applicant]
US 20130138264A1 · Hoshizaki et al. · 2013 [cited by applicant]
US 20140114561A1 · Pakzad et al. · 2014 [cited by applicant]
US 20150022640A1 · Metzler et al. · 2015 [cited by applicant]
US 20150331111A1 · Newman et al. · 2015 [cited by applicant]
US 20160140729A1 · Soatto et al. · 2016 [cited by applicant]
US 20180192035A1 · Dabeer et al. · 2018 [cited by applicant]
US 20180315209A1 · Murphy-Chutorian et al. · 2018 [cited by applicant]
US 20190041858A1 · Bortoff et al. · 2019 [cited by applicant]
US 20190347783A1 · Salgian et al. · 2019 [cited by applicant]
US 20200217666A1 · Zhang et al. · 2020 [cited by applicant]
US 20210213973A1 · Pena et al. · 2021 [cited by applicant]
US 20210247781A1 · Liu et al. · 2021 [cited by applicant]
US 20220026920A1 · Afrouzi et al. · 2022 [cited by applicant]
US 20220066456A1 · Afrouzi et al. · 2022 [cited by applicant]
US 20220217499A1 · Hasecke et al. · 2022 [cited by applicant]
US 20220277164A1 · Malayath · 2022 [cited by applicant]
US 20220319014A1 · Meitav · 2022 [cited by examiner]
US 20220398775A1 · Streem et al. · 2022 [cited by applicant]
US 20220404830A1 · Jobanputra et al. · 2022 [cited by applicant]
US 20220414919A1 · Tyagi et al. · 2022 [cited by applicant]
US 20230003188A1 · Vermillion et al. · 2023 [cited by applicant]
US 20230052288A1 · Mani et al. · 2023 [cited by applicant]
US 20230062186A1 · Measel et al. · 2023 [cited by applicant]
US 20230103297A1 · Savikovsky et al. · 2023 [cited by applicant]
US 20230105428A1 · Brown · 2023 [cited by examiner]
US 20230392944A1 · Kimia · 2023 [cited by applicant]
CN 108694882B · 2020 [cited by applicant]
CN 112200863A · 2021 [cited by examiner]
CN 115479602A · 2022 [cited by applicant]
WO WO2021242809A1 · 2021 [cited by examiner]
WO WO2024112351A1 · 2024 [cited by examiner]
Bancroft et al., “Data Fusion Algorithms for Multiple Inertial Measurement Units”, Article in Sensors (www.mdpi. com/journal/sensors, doi: 10.3390/s110706771 ), Jun. 29, 2011, pp. 6772-6798, 11(7), MDPI, Basel, Switzerl… [cited by applicant]
Company-Corcoles et al., “MSC-VO: Exploiting Manhattan and Structural Constraints for Visual Odometry”, Article in IEEE Robotics And Automation Letters, Jan. 13, 2022, pp. 2803-2810, 7(2), Institute of Electrical and El… [cited by applicant]
Corke, “Roll-pitch-yaw angles”, blog post on PeterCorke.com (https://petercorke.com/robotics/roll-pitch-yaw-angles/), Aug. 4, 2018, pp. 1-2, PeterCorke.com website by ZEPHYMEDIA (https://www.zephyrmedia.com.au/), Bundal… [cited by applicant]
Dynamics, “Rigid bodies”, webpage from Dynamics reference, Dec. 12, 2021, pp. 1-6, University of Illinois, Urbana-Champaign, Il, USA. [cited by applicant]
Eberly, “Approximations to Rotation Matrices and Their Derivatives”, web publication by Geometric Tools (https:// www.geometrictools.com/), Aug. 12, 2020, pp. 1-39, Geometric Tools, Redmond, Wa, USA. [cited by applicant]
Freescale Semiconductor, “Freescale Sensor Fusion for Kinetis Product Development Kit User Guide”, User Gude by Freescale Semiconductor, Sep. 1, 2014, pp. 1-46, Freescale Semiconductor, Austin, Tx, USA. [cited by applicant]
Freescale Semiconductor, “Freescale Sensor Fusion Library for Kinetis”, Datasheet by Freescale Semiconductor, Sep. 1, 2014, pp. 1-44, Freescale Semiconductor, Austin, Tx, USA. [cited by applicant]
Hou, “Modeling Inertial Sensors Errors Using Allan Variance”, Master thesis, Department of Geomatics Engineering, University of Calgary, Sep. 1, 2004, pp. 1-147, Department of Geomatics Engineering, University of Calgar… [cited by applicant]
Hover et al., “Rate of Change of Euler Angles”, Section 9.4.1 of MIT Courseware, System Design for Uncertainty (https://eng.libretexts.org/@go/page/47274), Sep. 6, 2021, pp. 1-1, LibreTexts Engineering (online education… [cited by applicant]
Li et al., “Accelerometer-Based Gyroscope Drift Compensation Approach in a Dual-Axial Stabilization Platform”, Article in Electronics (doi: 10.3390/electronics8050594), May 27, 2019, pp. 1-12, 8(594), MDPI (www.mdpi.com… [cited by applicant]
Malvezzi et al, “Train Position and Speed Estimation by Integration of Odometers and IMUs”, paper in 9th World Congress on Railway, May 22, 2011, pp. 22-26, 9th World Congress on Railway, Lille, France. [cited by applicant]
Mathworks, “Orientation from accelerometer and gyroscope readings”, webpage on mathworks.com (https://www.mathworks.com/help/fusion/ref/imufilter-system-object.html), Sep. 24, 2021, pp. 1-10, WathWorks, Natick, MA, USA. [cited by applicant]
Paina et al, “Experimental comparison of Kalman and complementary filter for attitude estimation”, Paper in conference XII Argentine Symposium on Technology (AST 2011)(XL JAIIO), Aug. 29, 2011, pp. 1-11, Cordoba, Argent… [cited by applicant]
Pedley et al, “Aerospace, Android and Windows 8 Coordinate Systems (orientation.c)”, Technical Note from Freescale Semiconductor, Dec. 1, 2014, pp. 1-27, Freescale Semiconductor, Austin, TX, USA. [cited by applicant]
Pedley et al, “Basic Kalman Filter Theory”, Technical Note from Freescale Semiconductor, Sep. 1, 2014, pp. 1-13, Freescale Semiconductor, Austin, TX, USA. [cited by applicant]
Pedley et al, “Calculation of Orientation Matrices from Sensor Data”, Technical Note from Freescale Semiconductor, Sep. 1, 2014, pp. 1-25, Freescale Semiconductor, Austin, TX, USA. [cited by applicant]
Pedley et al, “Freescale Sensor Fusion Kalman Filter (fusion.c)”, Technical Note from Freescale Semiconductor, Sep. 1, 2014, pp. 1-25, Freescale Semiconductor, Austin, TX, USA. [cited by applicant]
Pedley et al, “Magnetic Calibration (magnetic.c)”, Technical Note from Freescale Semiconductor, Sep. 1, 2014, pp. 1-27, Freescale Semiconductor, Austin, TX, USA. [cited by applicant]
Pedley et al, “Matrix and Vector Algebra (matrix.c)”, Technical Note from Freescale Semiconductor, Sep. 1, 2014, pp. 1-15, Freescale Semiconductor, Austin, TX, USA. [cited by applicant]
Pedley et al, “Quaternion Algebra and Rotation Quaternions”, Technical Note from Freescale Semiconductor, Sep. 1, 2014, pp. 1-17, Freescale Semiconductor, Austin, TX, USA. [cited by applicant]
Pedley et al, “Trigonometry Approximations (approximations.c)”, Technical Note from Freescale Semiconductor, Sep. 1, 2014, pp. 1-6, Freescale Semiconductor, Austin, TX, USA. [cited by applicant]
Pedley, “Tilt Sensing Using a Three-Axis Accelerometer”, Technical Document from Freescale Semiconductor, Mar. 1, 2013, pp. 1-22, Freescale Semiconductor, Austin, TX, USA. [cited by applicant]
Pizarro et al., “Automatic floor plan analysis and recognition”, Article in Automation in Construction, Aug. 1, 2022, pp. 1-20, 140, 104348, Department of Computer Science, Department of Civil Engineering, Universidad d… [cited by applicant]
Rasmussen et al., “Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning series)”, Appendix B of Book, ISBN: 026218253X, Nov. 23, 2005, pp. 207-219, The MIT Press, Boston, MA, USA. [cited by applicant]
Riley Jr, “Handbook of Frequency Stability Analysis”, Publication by Hamilton Technical Services (www.lulu.com as ID # 508588 ), Mar. 7, 2007, pp. 1-158, Hamilton Technical Services, Beaufort, SC, USA. [cited by applicant]
Roetenberg et al, “Compensation of Magnetic Disturbances Improves Inertial and Magnetic Sensing of Human Body Segment Orientation”, IEEE Transactions on Neural Systems and Rehabilitation Engineering, Sep. 12, 2005, pp. … [cited by applicant]
Salmony, “IMU Attitude Estimation”, webpage on philsal.co.uk, Sep. 24, 2021, pp. 1-13, city unknown, UK. [cited by applicant]
Slabaugh, “Computing Euler angles from a rotation matrix”, Article, Aug. 6, 2000, pp. 1-7, Queen Mary University of London, London, UK. [cited by applicant]
Sorkine-Hornung et al, “Least-Squares Rigid Motion Using SVD”, publication of Department of Computer Science, ETH Zurich, Jan. 16, 2017, pp. 1-5, ETH Zurich, Zurich, Switzerland. [cited by applicant]
Stanley, “Statistics gathered from build #420 of the sensor fusion library”, Technical Note from Freescale Semiconductor, Sep. 1, 2014, pp. 1, Freescale Semiconductor, Austin, TX, USA. [cited by applicant]
St Microelectronics, “iNEMO inertial module: 3D accelerometer, 3D gyroscope, 3D magnetometer”, Datasheet (DocID025715 Rev 3), Mar. 1, 2015, pp. 1-72, St Microelectronics, Santa Clara, CA, USA. [cited by applicant]
TDK, “Premium Performance 6-Axis Motion Tracking IMU”, TDK Document No. DS-000367 , Revision: 1.0, Rev. Date: Mar. 2, 2020 (www.invensense.com), Mar. 2, 2020, pp. 1-102, TDK Corporation, San Jose, CA, USA. [cited by applicant]
Timmer et al, “On Generating Power Law Noise”, Artcile in Astronomy and Astrophysics, Aug. 1, 1995, pp. 707-710, v. 300, EDP Sciences, Les Ulis, France. [cited by applicant]
Titterton et al., “Strapdown Inertial Navigation Technology”, Chapter 3 of Book, ISBN: 0863413587, Oct. 29, 2004, pp. 17-57, Institution of Engineering and Technology, Stevenage, UK. [cited by applicant]
Vagner, “MEMS Gyroscope Performance Comparison Using Allan Variance Method”, Doctoral Thesis in Proceedings of the 17th Conference Student EEICT, Jan. 1, 2011, pp. 199-203, Faculty of Electrical Engineering and Communic… [cited by applicant]
Welch et al, “An Introduction to the Kalman Filter”, Technical Report TR 95-041, Jul. 24, 2006, pp. 1-16, Department of Computer Science, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA. [cited by applicant]
Wikipedia, “Gauss-Markov process”, Wikipedia page, Jul. 5, 2023, pp. 1-2, Wikipedia Foundation Inc., San Francisco, CA, USA. [cited by applicant]
Woodman, “An introduction to inertial navigation”, Technical Report published by the University of Cambridge (ISSN 1476-2986, No. UCAM-CL-TR-696), Aug. 1, 2007, pp. 1-37, University of Cambridge, Cambridge, UK. [cited by applicant]
Yasin, “GyroBias Math Review”, Article, pp. 1-4, MindFolds Technology LLP, Bangalore, India. [cited by applicant]
Zhang et al, “Allan Variance Analysis on Error Characters of MEMS Inertial Sensors for an FPGA-based PS/INS System”, Artcile in Proceedings of the International Symposium on GPS/GNNS, Nov. 11, 2008, pp. 127-133, Univers… [cited by applicant]
PCT-US2025-024903_IntSrchRprt_672025009061996. [cited by applicant]
PCT-US2025-024903_WO_672025009062032. [cited by applicant]