IP Library › Granted Patent US 12,193,761
Granted Patent B2
US 12,193,761 · App. 18/096,989 · Granted Jan 14, 2025

Systems and methods for augmented reality based surgical navigation

Inventor: Stephen B. Murphy (Winchester, MA)
A61B34/20A61B34/10A61B90/90G02B27/017G06T19/003G06T19/006A61B1/00193A61B2034/102A61B2034/105A61B2034/107A61B2034/2051A61B2034/2055A61B2034/2068G02B2027/0174G02B2027/0178G06T2210/41G06T2219/008
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,193,761
App. No.
18/096,989
Filed
Jan 13, 2023
Granted
Jan 14, 2025
Kind
B2
Art Unit
2612
USPC
606/130
Abstract

The present disclosure involves object recognition as a method of registration, using a stereoscopic camera on Augmented Reality (AR) glasses or an endoscope as the image capture technology. Exemplary objects include surgical tools, anatomical components or features, such as bone or cartilage, etc. By detecting just a portion of the object in the image data of the surgical scene, the present disclosure may register and track a portion of the patient's anatomy, such as the pelvis, the knee, etc. The present disclosure also optionally displays information on the AR glasses themselves, such as the entire pelvis, the femur, the tibia, etc. The present disclosure may include combinations of the foregoing features, and may eliminate the need for electromagnetic, inertial, or infrared stereoscopic tracking as the tracking technology.

Claims (63)

1. A computer-implemented method comprising:

recognizing a registration and tracking device as docked to a portion of a patient's pelvis, the recognizing including tracking a location of the registration and tracking device, wherein the recognizing and the tracking performs a registration of the patient's pelvis relative to the registration and tracking device;

receiving files for presenting holograms of one or more surgical tools and at least one implant at determined locations relative to a coordinate system for the patient's pelvis;

receiving a transformation matrix for determining orientations and positions of the holograms relative to a coordinate system for the registration and tracking device or to the coordinate system for the patient's pelvis;

recognizing a tracker attached to the patient's pelvis, the recognizing the tracker including tracking a location of the tracker;

transferring the registration of the patient's pelvis from the registration and tracking device to the tracker;

removing the registration and tracking device from the portion of the patient's pelvis; and

utilizing the transformation matrix and the tracker to present the holograms anchored at the determined locations.

2. The computer-implemented method of claim 1 wherein

the registration and tracking device includes one or more codes, and

the recognizing the registration and tracking device includes detecting the one or more codes.

3. The computer-implemented method of claim 2 wherein the one or more codes expose one or more spatial coordinate systems.

4. The computer-implemented method of claim 3 wherein the transformation matrix includes a transform from a pelvic coordinate system to at least one of the one or more spatial coordinate system exposed by the one or more codes.

5. The computer-implemented method of claim 2 wherein

the tracker includes one or more recognizable tracker codes, and

the recognizing the tracker includes detecting the one or more tracker codes.

6. The computer-implemented method of claim 1 wherein the recognizing the registration and tracking device, the recognizing the tracker, and the utilizing the transformation matrix and the tracker to present the holograms are performed by an augmented reality (AR) head-mounted device (HMD).

7. A computer-implemented method comprising:

recognizing a tracker randomly attached to a patient's pelvis;

recognizing one or more digitizing probes;

determining locations for a plurality of points on a surface of the patient's pelvis, the plurality of points designated by the one or more digitizing probes during the recognizing the one or more digitizing probes;

utilizing the locations determined for the plurality of points on the surface of the patient's pelvis to perform a registration of the patient's pelvis relative to the tracker;

receiving files for presenting holograms of one or more surgical tools and at least one implant at determined locations relative to a coordinate system for the patient's pelvis;

determining orientations and positions of the holograms relative to the patient's pelvis based on the registration of the patient's pelvis relative to the tracker; and

utilizing the tracker to present the holograms anchored at the determined locations.

8. The computer-implemented method of claim 7 wherein the registration performed for the patient's pelvis includes determining the coordinate system for the patient's pelvis.

9. The computer-implemented method of claim 7 wherein the recognizing the tracker, the recognizing the one or more digitizing probes, the receiving the files, and the utilizing the tracker to present the holograms are performed by an augmented reality (AR) head-mounted device (HMD).

10. The computer-implemented method of claim 7 wherein

the tracker includes one or more codes, and

the recognizing the tracker includes detecting the one or more codes.

11. The computer-implemented method of claim 10 wherein

the one or more codes expose one or more spatial coordinate systems, and

the utilizing the tracker to present the holograms includes utilizing the one or more spatial coordinate systems.

12. One or more non-transitory computer-readable media comprising program instructions for execution by one or more processors, the program instructions instructing the one or more processors to:

recognize a registration and tracking device as docked to a portion of a patient's pelvis, the recognize including tracking a location of the registration and tracking device, wherein the recognize and the tracking performs a registration of the patient's pelvis relative to the registration and tracking device;

receive files for presenting holograms of one or more surgical tools and at least one implant at determined locations relative to a coordinate system for the patient's pelvis;

receive a transformation matrix for determining orientations and positions of the holograms relative to a coordinate system for the registration and tracking device or to the coordinate system for the patient's pelvis;

recognize a tracker attached to the patient's pelvis, the recognizing the tracker including tracking a location of the tracker;

transfer the registration of the patient's pelvis from the registration and tracking device to the tracker; and

with the registration and tracking device removed from the portion of the patient's pelvis, utilize the transformation matrix and the tracker to present the holograms anchored at the determined locations.

13. The one or more non-transitory computer-readable media of claim 12 wherein

the registration and tracking device includes one or more codes, and

the recognize the registration and tracking device includes detecting the one or more codes.

14. The one or more non-transitory computer-readable media of claim 13 wherein the one or more codes expose one or more spatial coordinate systems.

15. The one or more non-transitory computer-readable media of claim 14 wherein the transformation matrix includes a transform from a pelvic coordinate system to at least one of the one or more spatial coordinate system exposed by the one or more codes.

16. The one or more non-transitory computer-readable media of claim 13 wherein

the tracker includes one or more recognizable tracker codes, and

the recognize the tracker includes detecting the one or more tracker codes.

17. One or more non-transitory computer-readable media comprising program instructions for execution by one or more processors, the program instructions instructing the one or more processors to:

recognize a tracker randomly attached to a patient's pelvis;

recognize one or more digitizing probes;

determine locations for a plurality of points on a surface of the patient's pelvis, the plurality of points designated by the one or more digitizing probes during the recognizing the one or more digitizing probes;

utilize the locations determined for the plurality of points on the surface of the patient's pelvis to perform a registration of the patient's pelvis relative to the tracker;

receive files for presenting holograms of one or more surgical tools and at least one implant at determined locations relative to a coordinate system for the patient's pelvis;

determine orientations and positions of the holograms relative to the patient's pelvis based on the registration of the patient's pelvis relative to the tracker; and

utilize the tracker to present the holograms anchored at the determined locations.

18. The one or more non-transitory computer-readable media of claim 17 wherein the registration performed for the patient's pelvis includes determining the coordinate system for the patient's pelvis.

19. The one or more non-transitory computer-readable media of claim 17 wherein

the tracker includes one or more codes, and

the recognize the tracker includes detecting the one or more codes.

20. The one or more non-transitory computer-readable media of claim 19 wherein

the one or more codes expose one or more spatial coordinate systems, and

the utilize the tracker to present the holograms includes utilizing the one or more spatial coordinate systems.

Continuity (5)
Continuation 16888048 · May 29, 2020
Provisional Application 63000690 · Mar 27, 2020
Provisional Application 62913451 · Oct 10, 2019
Provisional Application 62853991 · May 29, 2019
Related Publication 20230149099A1 · May 18, 2023
References Cited (88)
US 8267938B2 · Murphy · 2012 [cited by applicant]
US 8986309B1 · Murphy · 2015 [cited by applicant]
US 10368947B2 · Lang · 2019 [cited by applicant]
US 10405927B1 · Lang · 2019 [cited by applicant]
US 10475244B2 · Cvetko et al. · 2019 [cited by applicant]
US 10535151B2 · Bleyer et al. · 2020 [cited by applicant]
US 10546423B2 · Jones et al. · 2020 [cited by applicant]
US 10603113B2 · Lang · 2020 [cited by applicant]
US 10869727B2 · Yanof et al. · 2020 [cited by applicant]
US 11638613B2 · Murphy · 2023 [cited by examiner]
US 20040152970A1 · Hunter et al. · 2004 [cited by applicant]
US 20090306679A1 · Murphy · 2009 [cited by applicant]
US 20110069867A1 · Buehner et al. · 2011 [cited by applicant]
US 20130310963A1 · Davison · 2013 [cited by applicant]
US 20140081659A1 · Nawana et al. · 2014 [cited by applicant]
US 20140276002A1 · West et al. · 2014 [cited by applicant]
US 20140282008A1 · Verard et al. · 2014 [cited by applicant]
US 20140358151A1 · Murphy et al. · 2014 [cited by applicant]
US 20150248793A1 · Abovitz et al. · 2015 [cited by applicant]
US 20150250450A1 · Thomas et al. · 2015 [cited by applicant]
US 20160143699A1 · Tanji · 2016 [cited by applicant]
US 20160191887A1 · Casas · 2016 [cited by applicant]
US 20160228193A1 · Moctezuma de la Barrera et al. · 2016 [cited by applicant]
US 20170196521A1 · Huang et al. · 2017 [cited by applicant]
US 20180168740A1 · Ryan et al. · 2018 [cited by applicant]
US 20180185100A1 · Weinstein et al. · 2018 [cited by applicant]
US 20180256256A1 · May et al. · 2018 [cited by applicant]
US 20180303558A1 · Thomas · 2018 [cited by applicant]
US 20180325618A1 · Justin et al. · 2018 [cited by applicant]
US 20190005724A1 · Pahud et al. · 2019 [cited by applicant]
US 20190025587A1 · Osterhout et al. · 2019 [cited by applicant]
US 20190167352A1 · Mahfouz · 2019 [cited by applicant]
US 20190170510A1 · Robinson · 2019 [cited by applicant]
US 20190285897A1 · Topliss et al. · 2019 [cited by applicant]
US 20200038117A1 · West et al. · 2020 [cited by applicant]
US 20200097119A1 · Pahud et al. · 2020 [cited by applicant]
US 20200110361A1 · Georgiou et al. · 2020 [cited by applicant]
US 20200111232A1 · Bleyer et al. · 2020 [cited by applicant]
US 20200375666A1 · Murphy · 2020 [cited by applicant]
US 20210100632A1 · Paradis · 2021 [cited by examiner]
US 20210106386A1 · Lang · 2021 [cited by examiner]
US 20210192759A1 · Lang · 2021 [cited by applicant]
US 20210290316A1 · Ng · 2021 [cited by examiner]
US 20230165641A1 · Kang · 2023 [cited by examiner]
CN 101170961A · 2008 [cited by applicant]
CN 103211655A · 2013 [cited by applicant]
FR 3088005A1 · 2020 [cited by applicant]
JP 2011502626A · 2011 [cited by applicant]
WO WO2018063528A1 · 2018 [cited by applicant]
WO WO2018078723A1 · 2018 [cited by applicant]
WO WO2018141787A1 · 2018 [cited by applicant]
WO WO2018165767A1 · 2018 [cited by applicant]
WO WO2018195529A1 · 2018 [cited by applicant]
WO WO2018203304A1 · 2018 [cited by applicant]
WO WO2020099268A1 · 2020 [cited by applicant]
WO WO2021225840A1 · 2021 [cited by applicant]
“FDA Letter to AugmedicsLtd.,” FDA U.S. Food & Drug Administration, Dec. 20, 2019, pp. 1-11. [cited by applicant]
“Notification of Transmittal of the International Search Report and the Written Opinion of the International Searching Authority, or the Declaration,” International Filing Date: Mar. 27, 2023, International Application … [cited by applicant]
U.S. Appl. No. 18/126,634, Filed: Mar. 27, 2023 by Stephen B. Murphy, M.D. for Systems and Methods for Planning Screw Lengths and Guiding Screw Trajectories During Surgery, pp. 1-158. [cited by applicant]
Abe, Yuichiro, et al., “A Novel 3D Guidance System Using Augmented Reality for Percutaneous Vertebroplasty,” American Association of Neurological Surgeons, AANS, Journal of Neurosurgery: Spine, vol. 19, Issue 4, Oct. 20… [cited by applicant]
An, Yong, et al., “A Novel Registration Method for Image-Guided Neurosurgery System Base on Stereo Vision,” Bio-Medical Materials and Engineering, IOS Press, vol. 26, Sep. 2015, pp. S967-S973. [cited by applicant]
Bamji, Cyrus S., et al., “5.8 1Mpixel 65nm BSI 320MHz Demodulated TOF Image Sensor with 3.5 μm Global Shutter Pixels and Analog Binning,” IEEE, IEEE International Solid-state Circuits Conference, ISSCC2018, Session 5, I… [cited by applicant]
Bichlmeier, Christoph, et al., “Contextual Anatomic Mimesis: Hybrid In-Situ Visualization Method for Improving Multi-Sensory Depth Perception in Medical Augmented Reality,” IEEE, 2007 6th IEEE and ACM International Symp… [cited by applicant]
Birkfellner, Wolfgang, et al. “Stereoscopic Visualization in the Varioscope AR: A See-Through Head-Mounted Display for Surgical Navigation,” Proceedings of SPIE, Medical Imaging 2002: Visualization, Image-Guided Procedu… [cited by applicant]
Chen, Long, et al., “Optimization of Virtual and Real Registration Technology Based on Augmented Reality in a Surgical Navigation System,” Research, BMC, BioMedical Engineering OnLine, vol. 19, No. 1, Jan. 8, 2020, pp. … [cited by applicant]
Chen, Xiaojun, et al., “Development of a Surgical Navigation System Based on Augmented Reality Using an Optical See-Through Head-Mounted Display,” Elsevier Inc., Elsevier, ScienceDirect, Journal of Biomedical Informatic… [cited by applicant]
“Coordinate Systems,” Microsoft, Jun. 11, 2019, pp. 1-6. [cited by applicant]
Frantz, Taylor, et al., “Augmenting Microsoft's Hololens with Vuforia Tracking for Neuronavigation,” Healthcare Technology Letters, vol. 5, Issue. 5, Oct. 4, 2018, pp. 221-225. [cited by applicant]
Liebmann, Florentin, et al., “Pedicle Screw Navigation using Surface Digitization on the Microsoft HoloLens,” original Article, CARS, Springer, International Journal of Computer Assisted Radiology and Surgery, vol. 14, … [cited by applicant]
Lin, Yen-Kun, et al., “A Novel Denal Implant Guided Surgery Based on Integration of Surgical Template and Augmented Reality,” Wiley Periodicals, Inc., Clinical Implant Dentistry and Related Research, Accuracy of a Compu… [cited by applicant]
Linte, Cristian A., et al., “On Mixed Reality Environments for Minimally Invasive Therapy Guidance: Systems Architecture, Success and Challenges in Their Implementation from Laboratory to Clinic,” Elsevier Ltd, National… [cited by applicant]
Liu, He, et al., “Augmented Reality Based Navigation for Computer Assisted Hip Resurfacing: A Proof of Concept Study,” Medical Robotics, Biomedical Engineering Society, BMES, Annals of Biomedical Engineering, vol. 46, N… [cited by applicant]
“Locatable Camera,” Locatable camera-Mixed Reality, Microsoft Docs, Microsoft, <https://docs.microsoft.com/en-us/windows/mixed-reality/locatable-camera>, Jun. 12, 2019, pp. 1-6. [cited by applicant]
Lovo, Eduardo, M.D., et al., “A Novel, Inexpensive Method of Image Coregistration for Applications in Image-Guided Surgery Using Augmented Reality,” Methodology Improvements, Neurosurgery, Operative Neurosurgery 2, vol.… [cited by applicant]
Mahvash, Mehran, et al., “A Novel Augmented Reality System of Image Projection for Image-Guided Neurosurgery,” Technical Note-Brain Tumors, Springer-Verlag Wien, Acta Neurochir, vol. 155, Mar. 15, 2013, pp. 943-947. [cited by applicant]
Mirota, Daniel J., et al., “Vision-Based Navigation in Image-Guided Interventions,” Annual Reviews, The Annual Review of Biomedical Engineering, vol. 13, May 10, 2011, pp. 297-319. [cited by applicant]
“Notification of Transmittal of the International Search Report and the Written Opinion of the International Searching Authority, or the Declaration,” International Filing Date: May 29, 2020, International Application N… [cited by applicant]
“QR Code Tracking,” Microsoft, Jun. 11, 2019, pp. 1-7. [cited by applicant]
Schönberger, Johannes, “Hololens as a Tool for Computer Vision Research,” ECCV 2018, Microsoft, Sep. 2018, pp. 1-31. [cited by applicant]
“Spatial Anchors,” Microsoft, May 16, 2019, pp. 1-3. [cited by applicant]
“Spatial Mapping,” Microsoft, Mar. 26, 2019, pp. 1-9. [cited by applicant]
Tabrizi Besharati, Leila, et al., Augmented Reality-Guided Neurosurgery: Accuracy and Intraoperative Application of an Image Projection Technique, Clinical Article, AANS, JNS, J Neurosurg, vol. 123, Jul. 2015, pp. 205-2… [cited by applicant]
Vavra, P., et al., “Recent Development of Augmented Reality in Surgery: A Review,” Review Article, Hindawi, Journal of Healthcare Engineering, vol. 2017, Article ID 4574172, Aug. 21, 2017, pp. 1-10. [cited by applicant]
“Voice Input,” Microsoft, Jun. 11, 2019, pp. 1-5. [cited by applicant]
Wang, Huixiang, et al., “Precision Insertion of Percutaneous Sacroiliac Screws using a Novel Augmented Reality-Based Navigation System: A Pilot Study,” Springer, SICOT aisbl, International Orthopaedics (SICOT), vol. 40,… [cited by applicant]
“Chinese Search Report for Chinese Application No. 202080054882.3”, dated Mar. 31, 2024, pp. 1-3. [cited by applicant]
English Translation of Japanese Office Action, from Japanese Patent Office for Japanese Patent Application No. JP 2021-570,328, dated Jan. 9, 2024, pp. 1-4. [cited by applicant]
Japanese Office Action, from Japanese Patent Office for Japanese Patent Application No. JP 2021-570,328, dated Jan. 9, 2024, pp. 1-4. [cited by applicant]