IP Library › Granted Patent US 12,467,720
Granted Patent B2
US 12,467,720 · App. 17/501,824 · Granted Nov 11, 2025

High-precision infantry training system (HITS)

Inventors: Stephen P. DelMarco (North Andover, MA); Simone B. Bortolami (Belmont, MA); Helen F. Webb (Malden, MA)
Assignee: BAE Systems Information and Electronic Systems Integration Inc.
F41G3/142F41G3/26
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,467,720
App. No.
17/501,824
Granted
Nov 11, 2025
Kind
B2
Abstract

A method of determining a point-of-impact of a ballistic projectile comprising: within a predetermined site using a weapon comprising a barrel, a camera, and an IMU configured to provide data concerning barrel attitude: detecting a firing event associated with the weapon; obtaining metadata associated with the weapon; determining estimated pointing angles of the weapon; obtaining a reference image of the site; culling portions of the reference image not associated with the estimated pointing angles, creating a culled reference image; projecting the culled reference image onto an image plane; obtaining an image from the weapon-mounted camera, the image being centered on the pointing direction weapon at the time of firing; registering the image from the weapon-mounted camera with the culled reference image; and calculating the true pointing angles of the weapon based on the alignment of the image obtained by the weapon-mounted camera with the culled reference image.

Claims (57)

1 . A method of determining a point-of-impact of a ballistic projectile, the method comprising:

within a predetermined site and using a weapon comprising a barrel, a weapon-mounted camera, and a weapon-mounted inertial measurement unit configured to provide data relating to weapon barrel attitude:

detecting a firing event associated with the weapon;

obtaining metadata associated with the weapon;

determining estimated pointing angles of the weapon;

obtaining a reference image of the predetermined site;

culling portions of the reference image not associated with the estimated pointing angles of the weapon, creating a culled reference image;

projecting the culled reference image onto an image plane;

obtaining an image from the weapon-mounted camera, the image being centered on a pointing direction the weapon at the time of firing;

registering the image from the weapon-mounted camera with the culled reference image; and

calculating true pointing angles of the weapon based on an alignment of the image obtained by the weapon-mounted camera with the culled reference image.

2 . The method of claim 1 wherein the firing event associated with the weapon is detected using an accelerometer mounted to the weapon that is configured to detect a recoil of the weapon.

3 . The method of claim 1 wherein the weapon further comprises a weapon-mounted magnetometer configured to provide data relating to weapon barrel attitude.

4 . The method of claim 1 wherein the weapon further comprises a Global Positioning System (GPS) receiver configured to provide weapon position information.

5 . The method of claim 4 wherein weapon position and weapon barrel attitude information is used to create a view frustum which intersects with the reference image.

6 . The method of claim 1 wherein the metadata comprises a quaternion that characterizes an orientation of the weapon-mounted camera to the reference image.

7 . The method of claim 1 wherein the reference image is a 2D image generated from a 3D site model.

8 . The method of claim 7 wherein the 3D site model is selected from the group consisting of 3D light detection and ranging (LIDAR) point-cloud data, a faceted 3D model, and 3D model.

9 . The method of claim 1 wherein the weapon-mounted camera is selected from the group consisting of infrared cameras, electro-optical cameras, and visible wavelength cameras.

10 . A system for determining a point-of-impact of a ballistic projectile, the system comprising:

a weapon comprising a barrel, a weapon-mounted camera, and a weapon-mounted inertial measurement unit; and

a processor in communication with the weapon-mounted camera and the weapon-mounted inertial measurement device, the processor configured to:

detect a firing event associated with the weapon;

obtain metadata associated with the weapon;

determine estimated pointing angles of the weapon;

obtain a reference image of a site corresponding to a location of the weapon;

cull portions of the reference image not associated with the estimated pointing angles of the weapon, creating a culled reference image;

project the culled reference image onto an image plane;

obtain an image from the weapon-mounted camera;

register the image from the weapon-mounted camera with the culled reference image; and

calculate true pointing angles of the weapon based on an alignment of the image obtained by the weapon-mounted camera with the culled reference image,

wherein the weapon-mounted camera is centered on a pointing direction of the weapon, and

wherein the weapon-mounted inertial measurement unit is configured to provide data relating to weapon barrel attitude to the processor.

11 . The system of claim 10 wherein the firing event associated with the weapon is detected using an accelerometer mounted to the weapon that is configured to detect a recoil of the weapon.

12 . The system of claim 10 wherein the weapon further comprises a weapon-mounted magnetometer configured to provide data relating to weapon barrel attitude and wherein the weapon-mounted magnetometer is in communication with the processor.

13 . The system of claim 10 wherein the weapon further comprises a Global Positioning System (GPS) receiver configured to provide weapon position information and wherein the GPS receiver is in communication with the processor.

14 . The system of claim 13 wherein weapon position and weapon barrel attitude information is used by the processor to create a view frustum which intersects with the reference image.

15 . The system of claim 10 wherein the metadata associated with the weapon comprises a quaternion that characterizes an orientation of the weapon-mounted camera to the reference image.

16 . The system of claim 10 wherein the reference image is a 2D image generated from a 3D site model.

17 . The system of claim 16 wherein the 3D site model is selected from the group consisting of 3D light detection and ranging (LIDAR) point-cloud data, a faceted 3D model, and 3D model.

18 . The system of claim 10 wherein the weapon-mounted camera is selected from the group consisting of infrared cameras, electro-optical cameras, and visible wavelength cameras.

19 . An apparatus for determining a point-of-impact of a ballistic projectile, the apparatus comprising:

a camera, an inertial measurement unit; and

a processor in communication with the camera and the inertial measurement device, the processor configured to:

detect a firing event associated with a weapon;

obtain metadata associated with the weapon;

determine estimated pointing angles of the camera and/or inertial measurement unit;

obtain a reference image of a site corresponding to a location of the apparatus;

cull portions of the reference image not associated with the estimated pointing angles of the camera, creating a culled reference image;

project the culled reference image onto an image plane;

obtain an image from the camera;

register the image from the camera with the culled reference image; and

calculate the true pointing angles of the camera based on an alignment of the image obtained by the camera with the culled reference image,

wherein the camera and inertial measurement unit are configured to be mounted on the weapon,

wherein the camera is configured to be centered on a pointing direction of the weapon, once mounted thereto, and

wherein the weapon-mounted inertial measurement unit is configured to provide attitude data to the processor.

20 . The apparatus of claim 19 further comprising a Global Positioning System (GPS) receiver configured to provide position information, wherein position and attitude information is used to create a view frustum which intersects with the reference image, and wherein the metadata comprises a quaternion that characterizes the orientation of the camera to the reference image.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 18, 2021
From: DELMARCO, STEPHEN P.; BORTOLAMI, SIMONE B.; WEBB, HELEN F.
To: BAE SYSTEMS INFORMATION AND ELECTRONIC SYSTEMS INTEGRATION INC.
Reel/Frame 057815/0169 →
Continuity (1)
Related Publication 20250224207A1 · Jul 10, 2025
References Cited (58)
US 9766074B2 · Roumeliotis et al. · 2017 [cited by applicant]
US 10012504B2 · Roumeliotis et al. · 2018 [cited by applicant]
US 10107919B1 · Chapman et al. · 2018 [cited by applicant]
US 10151588B1 · Singh et al. · 2018 [cited by applicant]
US 10254118B2 · Roumeliotis et al. · 2019 [cited by applicant]
US 10325411B1 · Laney et al. · 2019 [cited by applicant]
US 10366549B1 · Mash · 2019 [cited by applicant]
US 10694148B1 · Li et al. · 2020 [cited by applicant]
US 10907971B2 · Roumeliotis et al. · 2021 [cited by applicant]
US 10929713B2 · Chiu et al. · 2021 [cited by applicant]
US 11466990B2 · Roumeliotis et al. · 2022 [cited by applicant]
US 11544161B1 · Yarlagadda et al. · 2023 [cited by applicant]
US 20040050240A1 · Greene · 2004 [cited by examiner]
US 20070070069A1 · Samarasekera et al. · 2007 [cited by applicant]
US 20100295942A1 · Jekel · 2010 [cited by examiner]
US 20110218733A1 · Hamza et al. · 2011 [cited by applicant]
US 20130130205A1 · Matthews · 2013 [cited by examiner]
US 20130314509A1 · Laine et al. · 2013 [cited by applicant]
US 20150286217A1 · Qian et al. · 2015 [cited by applicant]
US 20170329335A1 · DelMarco et al. · 2017 [cited by applicant]
US 20180091746A1 · Benser et al. · 2018 [cited by applicant]
US 20190051056A1 · Chiu et al. · 2019 [cited by applicant]
US 20190368877A1 · O'Shea et al. · 2019 [cited by applicant]
US 20210048821A1 · Bondurant et al. · 2021 [cited by applicant]
US 20220006557A1 · Pierson et al. · 2022 [cited by applicant]
WO 2016174659A1 · 2016 [cited by applicant]
Delmarco, S., Webb, H., and Tom, V., “Spectrally-Shaped Correlation with Application to Image Registration,” Proceedings of SPIE, vol. 10993, 2019. [cited by applicant]
Delmarco, S., Webb, H., and Tom, V., “A Progressive Refinement Approach to Aerial Image Registration Using Local Transform Perturbations,” Proceedings of the IEEE International Geoscience and Remote Sensing. Symposium, … [cited by applicant]
Delmarco, S., “Multi-Template Image Matching Using Alpha-Rooted Biquaternion Phase Correlation with Application to Logo Recognition,” Proceedings of SPIE, vol. 8063, 2011. [cited by applicant]
Delmarco, S., “Multiple Template-Based Image Matching Using Alpha-Rooted Quaternion Phase Correlation,” Proceedings of SPIE, vol. 7708, 2010. [cited by applicant]
Delmarco, S., “Logo Recognition Using Alpha-Rooted Phase Correlation in the Radon Transform Domain,” Proceedings of SPIE, vol. 7443, 2009. [cited by applicant]
Delmarco, S., Tom, V., Webb, H., and Lefebvre, D., “A Verification Metric for Multi-Sensor Image Registration,” Proceedings of SPIE, vol. 6567, 2007. [cited by applicant]
Delmarco, S., Webb, H., and Tom, V., “Automatic Spatial Accuracy Estimation for Correlation-Based Image Registration,” Proceedings of SPIE, vol. 10993, 2019. [cited by applicant]
“A Novel Approach for Fusing LIDAR and Visual Camera images in Unstructured environment” International Conference on Advanced Computing and Communication Systems (ICACCS—2017), Jan. 6-7, 2017, Coimbatore, India. [cited by applicant]
Anderson Lebbad, C. Nataraj “A Bayesian Algorithm for Vision Based Navigation of Autonomous Surface Vehicles” 2015 IEEE 7th International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE Conference on Ro… [cited by applicant]
Ben Mildenhall et al. “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis” 17 pages. [cited by applicant]
Cesar Debeunne et al. “A Review of Visual-LiDAR Fusion based Simultaneous Localization and Mapping” Published Apr. 7, 2020, retrieved from the internet www.mdpi.com/journal/sensors, Sensors 2020, 20, 2068; doi:10.3390/s… [cited by applicant]
Darshan Bhanushali et al. “LiDAR—Camera Fusion for 3D Object Detection” in Proc. IS&T Int'l. Symp. on Electronic Imaging: Autonomous Vehicles and Machines, 2020, pp. 257-1-257-9, https://doi.org/10.2352/ISSN.2470-1173.2… [cited by applicant]
Hyunggi Jo et al. “New Monte Carlo Localization Using Deep Initialization: A Three-Dimensional LiDAR and a Camera Fusion Approach” vol. 8 2020, School of Electrical and Electronic Engineering, Yonsei University, Seoul 0… [cited by applicant]
Jing Li et al. “OpenStreetMap-Based Autonomous Navigation for the Four Wheel-Legged Robot Via 3D-Lidar and CCD Camera” in IEEE Transactions on Industrial Electronics, vol. 69, No. 3, pp. 2708-2717, Mar. 2022, doi: 10.11… [cited by applicant]
Johannes Graeter et al. “LIMO: Lidar-Monocular Visual Odometry” 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain, 2018, pp. 7872-7879, doi: 10.1109/IROS.2018.8594394. [cited by applicant]
Mao Shan et al “Probabilistic Egocentric Motion Correction of Lidar Point Cloud and Projection to Camera Images for Moving Platforms” 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC),… [cited by applicant]
Naman Patel et al. “Sensor Modality Fusion with CNNs for UGV Autonomous Driving Indoor Environments” 8 pages. [cited by applicant]
Notice of Allowance of U.S. Appl. No. 18/474,817, mail date Jan. 8, 2025, 17 pages. [cited by applicant]
Qi Kong et al. “Outdoor real-time RGBD sensor fusion of stereo camera and sparse lidar” (2022) J. Phys.: Conf. Ser. 2234 012010, doi: 10.1088/1742-6596/2234/1/012010. [cited by applicant]
Qingqu Wang et al. “On SLAM Based on Monocular Vision and Lidar Fusion System” 2018 IEEE CSAA Guidance, Navigation and Control Conference (CGNCC), Xiamen, China, 2018, pp. 1-5, doi: 10.1109/GNCC42960.2018.9018987. [cited by applicant]
Qingquan Li et al. “A Sensor-Fusion Drivable-Region and Lane-Detection System for Autonomous Vehicle Navigation in Challenging Road Scenarios” IEEE Transactions on Vehicular Technology, vol. 63, No. 2, p. 540-555, Feb. … [cited by applicant]
Sara Fridovich-Keil et al. “Plenoxels: Radiance Fields without Neural Networks” 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 5491-5500, doi: 10.1109/CVPR526… [cited by applicant]
Sebastion Schneider et al. “Fusing Vision and LIDAR—Synchronization, Correction and Occlusion Reasoning” 2010 IEEE Intelligent Vehicles SymposiumUniversity of California, San Diego, CA, USA Jun. 21-24, 2010, 6 pages. [cited by applicant]
Shahram Moafipoor et al. “LiDAR/Camera Point & Pixel Aided Autonomous Navigation” LiDAR Magazine, vol. 7 No. 7, 7 pages, https://lidarmag.com/2017/10/25/lidarcamera-point-pixel-aided-autonomous-navigation/. [cited by applicant]
Shi-Sheng Huang et al. “Lidar-Monocular Visu9al Odometry using Point and Line Features” 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France, 2020, pp. 1091-1097, doi: 10.1109/ICRA40945.20… [cited by applicant]
Stephan J. Garbin et al. “FastNeRF: High-Fidelity Neural Rendering at 200FPS” 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 2021, pp. 14326-14335, doi: 10.1109/ICCV48922.2021.01… [cited by applicant]
Stephen Delmarco et al. “High-Precision Infantry Training System (HITS)” Proceedings of the 33rd International Technical Meeting of the Satellite Division of The Institute of Navigation (ION GNSS+ 2020), Sep. 2020, pp. … [cited by applicant]
Stephen Delmarco, Helen Webb, Victor Tom “Automatic Spatial Accuracy Estimation Correlation-based Image Registration” Proc. SPIE 10993, Mobile Multimedia/Image Processing, Security, and Applications 2019, 1099307 (May 1… [cited by applicant]
Stephen Delmarco, Helen Webb, Victor Tom “Spectrally-shaped Correlation With Application to Image Registration” Proc. SPIE 10993, Mobile Multimedia/Image Processing, Security, and Applications 2019, 1099306 (May 13, 201… [cited by applicant]
Xiangmo Zhao et al. “Fusion of 3D LIDAR and Camera Data for Object Detection in Autonomous Vehicle Applications” IEEE Sensors Journal, vol. 20, No. 9, May 1, 2020, 13 pages. [cited by applicant]
Yilin Zhang “LIDAR—camera deep fusion for end-to-end trajectory planning autonomous vehicle” 2022 J. Phys.: Conf. Ser. 2284 012006, retrieved from the internet: https://iopscience.iop.org/article/10.1088/1742-6596/2284/… [cited by applicant]
Young-Sik Shin et al. “Direct Visual SLAM using Sparse Depth for Camear-LiDAR System” 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, QLD, Australia, 2018, pp. 5144-5151, doi: 10.1109/ICR… [cited by applicant]