IP Library Granted Patent US 12,290,271
Granted Patent B2
US 12,290,271 · App. 17/685,345 · Granted May 6, 2025

System and method for location determination using movement between optical labels and a 3D spatial mapping camera

Inventors: Russell Todd Nevins (Las Vegas, NV); David Jon Backstein (Toronto, CA); Bradley H. Nathan (Toronto, CA)
Assignees: Russell Todd Nevins; David Jon Backstein; Bradley H. Nathan
A61B17/1764A61B34/20A61B90/39G02B27/017G06F3/011A61B2090/395
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,290,271
App. No.
17/685,345
Granted
May 6, 2025
Kind
B2
Abstract

A system determining a location for a surgical procedure, the system including a jig having a frame, a first marker fixed to the frame, wherein the first marker includes a scanable label, and a second marker moveably connected to the frame, such that the second marker can move positions independent of the frame, and wherein the second marker includes a scanable label. The system also includes a mixed reality headset configured to scan the scanable label of the first marker and the scanable label of the second marker, to provide location data to the mixed reality headset.

Claims (78)

1. A method of determining a location for a surgical procedure, comprising:

attaching a jig to a bone, wherein the jig includes:

a frame;

a first marker fixed to the frame, wherein the first marker includes a scanable label;

a second marker moveably connected to the frame, such that the second marker can move positions independent of the frame, and wherein the second marker includes a scanable label;

wherein the frame includes a slot configured to receive the second marker;

at least one locking pin configured to bias the frame against the second marker within the slot;

providing a 3D spatial mapping camera;

spatially mapping the jig and the bone using the 3D spatially mapping camera to identify a location for the surgical procedure;

scanning the scanable label of the first marker and the scanable label of the second marker to identify a location for the second marker; and

manipulate a position of the second marker to the identified location.

2. The method of claim 1 , further comprising:

providing a mixed reality headset configured to scan the scanable label of the first marker and the scanable label of the second marker, to provide location data to the mixed reality headset.

3. The method of claim 1 , wherein the first marker is fixed to the frame such that the first marker does not move independent of the frame.

4. The method of claim 1 , wherein the scanable label of the first marker includes data receivable by the mixed reality headset related to the jig.

5. The method of claim 1 , wherein the scanable label of the second marker includes data receivable by the mixed reality headset related to a determined position of the second marker.

6. The method of claim 1 , wherein the second marker includes a target, configured to identify a location to drill.

7. The method of claim 1 , wherein the second marker may move laterally, with respect to the frame, within the slot.

8. The method of claim 7 , wherein the slot is configured to receive the second marker with a friction fit engagement.

9. The method of claim 1 , further comprising a mounting plate fastened to the frame and configured to abut a bone.

10. The method of claim 1 , wherein the scanable label of the first marker is removable.

11. The method of claim 1 , wherein the scanable label of the second marker is removable.

12. A method of determining a location for a surgical procedure, comprising:

attaching a jig to a bone, wherein the jig includes:

a frame;

a first marker fixed to the frame, wherein the first marker includes a scanable label;

a second marker moveably connected to the frame, such that the second marker can move positions independent of the frame, and wherein the second marker includes a scanable label;

wherein the frame includes a slot configured to receive the second marker;

at least one locking pin, separate from the second marker, configured to bias the frame against the second marker within the slot;

providing a 3D spatial mapping camera;

providing a mixed reality headset configured to communicate with the 3D mapping camera;

spatially mapping the jig and the bone using the 3D spatially mapping camera to identify a location for the surgical procedure;

sending a determined location of the second marker from the 3D spatial mapping camera to the mixed reality headset, such that the mixed reality headset provides an image of the determined location for the second marker relative to the bone; and

manipulating a position of the second marker to the identified location.

13. The method of claim 12 , further comprising:

scanning the scanable label of the first marker and the scanable label of the second marker to identify the determined location for the second marker.

14. The method of claim 12 , wherein the first marker is fixed to the frame such that the first marker does not move independent of the frame.

15. The method of claim 12 , wherein the scanable label of the first marker includes data receivable by the mixed reality headset related to the jig.

16. The method of claim 12 , wherein the scanable label of the second marker includes data receivable by the mixed reality headset related to a determined position of the second marker.

17. The method of claim 12 , wherein the second marker includes a target, configured to identify a location to drill.

18. The method of claim 12 , wherein the second marker may move laterally, with respect to the frame, within the slot.

19. The method of claim 18 , wherein the slot is configured to receive the second marker with a friction fit engagement.

20. The method of claim 12 , further comprising a mounting plate fastened to the frame and configured to abut a bone.

21. A method of determining a location for a surgical procedure, comprising:

attaching a jig to a bone, wherein the jig includes:

a frame;

a first marker fixed to the frame, wherein the first marker includes a scanable label;

a second marker moveably connected to the frame, such that the second marker can move positions independent of the frame, and wherein the second marker includes a scanable label;

wherein the frame includes a slot configured to receive the second marker, the slot being defined by opposing interior surfaces;

at least one locking pin configured to bias said opposing interior surfaces of the slot against the second marker;

providing a mixed reality headset;

scanning the scanable label of the first marker and the scanable label of the second marker with the mixed reality headset to identify a location for the second marker using the mixed reality headset; and

manipulating a position of the second marker to the identified location.

22. The method of claim 21 , further comprising:

providing a 3D spatial mapping camera; and

spatially mapping the jig and the bone using the 3D spatially mapping camera to identify a location for the surgical procedure.

23. The method of claim 21 , wherein the first marker is fixed to the frame such that the first marker does not move independent of the frame.

24. The method of claim 21 , wherein the scanable label of the first marker includes data receivable by the mixed reality headset related to the jig.

25. The method of claim 21 , wherein the scanable label of the second marker includes data receivable by the mixed reality headset related to a determined position of the second marker.

26. The method of claim 21 , wherein the second marker includes a target, configured to identify a location to drill.

27. The method of claim 21 , wherein the second marker may move laterally, with respect to the frame, within the slot.

28. The method of claim 27 , wherein the slot is configured to receive the second marker with a friction fit engagement.

29. The method of claim 21 , further comprising a mounting plate fastened to the frame and configured to abut a bone.

30. A method of determining a location for a surgical procedure, comprising:

providing a 3D spatial mapping camera;

providing a mixed reality headset;

attaching a jig to a bone, wherein the jig includes:

a frame;

a first marker fixed to the frame, wherein the first marker includes a removable scanable label, wherein the first marker is fixed to the frame such that the first marker does not move independent of the frame and the scanable label of the first marker includes data receivable by the mixed reality headset related to the jig;

a second marker moveably connected to the frame, such that the second marker can move positions independent of the frame, and wherein the second marker includes a removable scanable label that includes data receivable by the mixed reality headset related to a determined position of the second marker and wherein the second marker also includes a target, configured to identify a location to drill;

wherein the frame includes a slot configured to receive the second marker, the slot being defined by opposing interior surfaces, said opposing interior surfaces being disposed in friction fit engagement with the second marker;

at least one locking pin configured to bias said opposing interior surfaces of the slot against the second marker

a mounting plate fastened to the frame and configured to abut a bone;

wherein the frame includes a slot configured to receive the second marker, wherein the second marker may move laterally, with respect to the frame, within the slot and wherein the slot is configured to receive the second marker and the slot being defined by opposing interior surfaces, said opposing interior surfaces being disposed in friction fit engagement with the second marker;

at least one locking pin configured to bias said opposing interior surfaces of the slot against the second marker

spatially mapping the jig and the bone using the 3D spatially mapping camera to identify a location for the surgical procedure;

scanning the scanable label of the first marker and the scanable label of the second marker with the mixed reality headset to identify a location for the second marker using the mixed reality headset; and

manipulating a position of the second marker to the identified location.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 2, 2025
From: NEVINS, RUSSELL TODD; BACKSTEIN, DAVID JON; NATHAN, BRADLEY H.
To: NEVINS, RUSSELL TODD; BACKSTEIN, DAVID JON; NATHAN, BRADLEY H.
Reel/Frame 070717/0409 →
Continuity (6)
Division 17486704 · Sep 27, 2021
Continuation 17373613 · Jul 12, 2021
Continuation 17169289 · Feb 5, 2021
Continuation 17030352 · Sep 23, 2020
Continuation 16994663 · Aug 17, 2020
Related Publication 20220183705A1 · Jun 16, 2022
References Cited (118)
US 7331929B2 · Morita et al. · 2008 [cited by applicant]
US 7812815B2 · Banerjee · 2010 [cited by applicant]
US 8123754B2 · Siebel · 2012 [cited by applicant]
US 8560047B2 · Haider et al. · 2013 [cited by applicant]
US 8876830B2 · Hodorek · 2014 [cited by examiner]
US 8954181B2 · MacLeod et al. · 2015 [cited by applicant]
US 8956165B2 · Kurenov · 2015 [cited by applicant]
US 9563266B2 · Banerjee · 2017 [cited by applicant]
US 9730713B2 · Park · 2017 [cited by applicant]
US 9861446B2 · Lang · 2018 [cited by examiner]
US 9892564B1 · Cvetko et al. · 2018 [cited by applicant]
US 9978141B2 · Stolka et al. · 2018 [cited by applicant]
US 9980780B2 · Lang · 2018 [cited by applicant]
US 10016243B2 · Esterberg · 2018 [cited by applicant]
US 10108266B2 · Banerjee · 2018 [cited by applicant]
US 10159530B2 · Lang · 2018 [cited by applicant]
US 10194990B2 · Amanatullah · 2019 [cited by examiner]
US 10220181B2 · Giap · 2019 [cited by applicant]
US 10241569B2 · Lanman · 2019 [cited by applicant]
US 10278777B1 · Lang · 2019 [cited by applicant]
US 10285765B2 · Sachs · 2019 [cited by applicant]
US 10286179B2 · Giap · 2019 [cited by applicant]
US 10292768B2 · Lang · 2019 [cited by applicant]
US 10368947B2 · Lang · 2019 [cited by applicant]
US 10401954B2 · Koker · 2019 [cited by applicant]
US 10405873B2 · Amiot · 2019 [cited by applicant]
US 10405927B1 · Lang · 2019 [cited by applicant]
US 10437335B2 · Daniels · 2019 [cited by applicant]
US 10437339B2 · Banerjee · 2019 [cited by applicant]
US 10602114B2 · Casas · 2020 [cited by applicant]
US 10672288B2 · Ribeira et al. · 2020 [cited by applicant]
US 10716643B2 · Justin et al. · 2020 [cited by applicant]
US 10888399B2 · Kopelman et al. · 2021 [cited by applicant]
US 10980601B2 · Yang et al. · 2021 [cited by applicant]
US 11045263B1 · Nevins et al. · 2021 [cited by applicant]
US 11172990B2 · Lang · 2021 [cited by applicant]
US 20050251030A1 · Azar · 2005 [cited by examiner]
US 20080183179A1 · Siebel · 2008 [cited by applicant]
US 20090163923A1 · Flett · 2009 [cited by examiner]
US 20140222462A1 · Shakil et al. · 2014 [cited by applicant]
US 20160191887A1 · Casas · 2016 [cited by examiner]
US 20170245781A1 · Kay · 2017 [cited by applicant]
US 20170258526A1 · Lang · 2017 [cited by examiner]
US 20170312032A1 · Amanatullah · 2017 [cited by examiner]
US 20170367766A1 · Mahfouz · 2017 [cited by applicant]
US 20170367771A1 · Tako et al. · 2017 [cited by applicant]
US 20180049622A1 · Ryan · 2018 [cited by examiner]
US 20180090029A1 · Fisher · 2018 [cited by applicant]
US 20180098813A1 · Nesichi · 2018 [cited by applicant]
US 20180116728A1 · Lang · 2018 [cited by applicant]
US 20180240276A1 · He et al. · 2018 [cited by applicant]
US 20180348876A1 · Banerjee · 2018 [cited by applicant]
US 20190038362A1 · Nash et al. · 2019 [cited by applicant]
US 20190076198A1 · Berend · 2019 [cited by applicant]
US 20190110842A1 · Lang · 2019 [cited by applicant]
US 20190142520A1 · VanDyken · 2019 [cited by applicant]
US 20190149797A1 · Casas · 2019 [cited by applicant]
US 20190216562A1 · Sachs · 2019 [cited by applicant]
US 20190262078A1 · Lang · 2019 [cited by applicant]
US 20190366030A1 · Giap et al. · 2019 [cited by applicant]
US 20190380792A1 · Poltaretskyi et al. · 2019 [cited by applicant]
US 20200000527A1 · Cazal · 2020 [cited by applicant]
US 20200037043A1 · Phillips et al. · 2020 [cited by applicant]
US 20200078100A1 · Weinstein et al. · 2020 [cited by applicant]
US 20200107003A1 · Phillips et al. · 2020 [cited by applicant]
US 20200275976A1 · McKinnon · 2020 [cited by examiner]
US 20200275988A1 · Johnson et al. · 2020 [cited by applicant]
US 20200302694A1 · Flexman et al. · 2020 [cited by applicant]
US 20200360093A1 · Khan et al. · 2020 [cited by applicant]
US 20200375666A1 · Murphy · 2020 [cited by examiner]
US 20210093329A1 · Poltaretskyi et al. · 2021 [cited by applicant]
US 20210093391A1 · Poltaretskyi et al. · 2021 [cited by applicant]
US 20210093413A1 · Poltaretskyi et al. · 2021 [cited by applicant]
US 20210228286A1 · Moghaddam et al. · 2021 [cited by applicant]
US 20210228308A1 · Berger et al. · 2021 [cited by applicant]
US 20210244481A1 · Jaramaz et al. · 2021 [cited by applicant]
US 20220047279A1 · Nevins et al. · 2022 [cited by applicant]
US 20220051483A1 · Nevins et al. · 2022 [cited by applicant]
CN 107430437A · 2017 [cited by applicant]
CN 110430809A · 2019 [cited by applicant]
CN 110431636A · 2019 [cited by applicant]
DE 10103922A1 · 2002 [cited by applicant]
DE 102015212352A1 · 2017 [cited by applicant]
EP 3113682A1 · 2017 [cited by applicant]
GB 2570758A · 2019 [cited by applicant]
JP 2020515891A · 2020 [cited by applicant]
WO 2007108776A2 · 2007 [cited by applicant]
WO 2012033739A2 · 2012 [cited by applicant]
WO 2015134953A1 · 2015 [cited by applicant]
WO 2017066373A1 · 2017 [cited by applicant]
WO 2018007091A1 · 2018 [cited by applicant]
WO 2018132804A1 · 2018 [cited by applicant]
WO 2018175971A1 · 2018 [cited by applicant]
WO 2019051080A1 · 2019 [cited by applicant]
WO 2019245870A1 · 2019 [cited by applicant]
WO 2020033568A2 · 2020 [cited by applicant]
WO 2020037308A1 · 2020 [cited by applicant]
WO 2020047051A1 · 2020 [cited by applicant]
WO 2020145826A1 · 2020 [cited by applicant]
WO 2021094354A1 · 2021 [cited by applicant]
Microsoft HoloLens & Mixed /Reality Healthcare Industry Deck, unknown author, at least as early as Oct. 14, 2019. [cited by applicant]
Kaluschke et al., HIPS—A Virtual Reality Hip Prosthesis Implantation Simulator, retrieved at https://www.reasearchgate.net/publication/327329265, upload date Sep. 3, 2018 DOI: 10.1109/VR.2018.8446370. [cited by applicant]
Vaughan et al., Does Virtual-Reality Training on Orthopaedic Simulators Improve Performance in the Operating Room? Science and Information Conference 2015, Jul. 28-30, 2015, London, UK; retrieved at https://www.research… [cited by applicant]
Patently Apple—Apple Reveals a Mixed Reality Headset that Uses a Direct Retinal Projector System With Holographic Lenses, retrieved at https://www.patentlyapple.com/patently-apple/2019/09/apple-reveals-a-mixed-reality-h… [cited by applicant]
Virtual Reality System Helps Surgeons, Reassures Patients, retrieved at https/medicalgiving.stanford.edu/news/virtual-reality-system-helps-surgeons-reassures-patients.html, retrieved date Oct. 24, 2019. [cited by applicant]
Immersive Touch Launches the First Virtual Reality Integrated Suite for Surgical Planning, retrieved at https://spinalnewsinternational.com/immersivetouch-virtual-reality-suite, dated Oct. 5, 2018. [cited by applicant]
Daley, Sam, The Cutting Edge: 10 Companies Bringing Virtual Reality & AR to the OR, retrieved at https://builtin.com/healthcare-technology/augmented-virtual-reality-surgery, dated Jul. 5, 2019. [cited by applicant]
Barad, Justin, Virtual and Augmented Reality Can Save Lives by Improving Surgeons' Training, retrieved at https://www.statnews.com/2019/08/16/virtual-reality-improve-surgeon-training, dated Aug. 16, 2019. [cited by applicant]
Levin et al., The Future of Virtual Reality in Ophthalmology Is Already Here, retrieved at https://www.aao.org/young-ophthalmologists/yo-info/article/future-of-virtual-reality-in-ophthalmology, dated Aug. 16, 2019. [cited by applicant]
Vaughan et al., A Review of Virtual Reality Based Training Simulators for Orthopaedic Surgery, retrieved at https://www.researchgate.net/publication/283727217, posted date Feb. 22, 2019, DOI: 10.1016/j.medengphy.2015.11… [cited by applicant]
LexInnova Patent Landscape Analysis, Virtual Reality, unknown author, copyright date of 2015. [cited by applicant]
Virtual & Augmented Reality Are You Sure it Isn't Real? Kathleen Boyle, CFA, Managing Editor, Citi GPS dated Oct. 2016. [cited by applicant]
New Apple patent filing shows a mixed reality headset that tracks your whole face, Jul. 22, 2019, (downloaded Jul. 1, 2020 at https://www.theverge.com/2019/7/22/20705158/apple-mixed-reality-headset-ar-glasses-patent-app… [cited by applicant]
“Augmented and virtual reality in surgery-the digital surgical environment: application, limitations and legal pitfalls,” accessed at https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5220044/, visited on Jul. 4, 2020. 12 pa… [cited by applicant]
“The impact of Web3D technologies on medical education and training,” Science Direct, accessed at https://www.sciencedirect.com/science/article/pii/S0360131505000825, visited on Jul. 4, 2020, 11 pages. [cited by applicant]
“Mixed Reality with Hololens: Where Virtual Reality Meets Augmented Reality in the Operating Room,” accessed at https://www.ingentaconnect.com/content/wk/prs/2017/00000140/00000005/art00063, visited on Jul. 4, 2020, 1 p… [cited by applicant]
“Virtual Reality Simulation in Neurosurgery: Technologies and Evolution,” Abstract, accessed at https://academic.oup.com/neurosurgery/article-abstract/72/suppl_1/A154/2417686, visited on Jul. 4, 2020, 2 pages. [cited by applicant]
Katanacho, Manuel, Wladimir De la Cadena, and Sebastian Engel. “Surgical navigation with QR codes: Marker detection and pose estimation of QR code markers for surgical navigation.” Current Directions in Biomedical Engin… [cited by applicant]