IP Library › Granted Patent US 12,446,969
Granted Patent B2
US 12,446,969 · App. 17/456,989 · Granted Oct 21, 2025

Robot mounted camera registration and tracking system for orthopedic and neurological surgery

Inventors: Benjamin Sexson (Laguna Hills, CA); Douglas B. Unis (Brooklyn, NY); Matthew Dicicco (Brooklyn, NY); Brian Jin (New York, NY)
Assignees: ICAHN SCHOOL OF MEDICINE AT MOUNT SINAI; MONOGRAM ORTHOPAEDICS, INC.
A61B34/20A61B34/30A61B2034/2057A61B2034/2068
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,446,969
App. No.
17/456,989
Granted
Oct 21, 2025
Kind
B2
Abstract

A surgical method includes, for example, tracking, via at least one camera attached to a robotic arm of a surgical robot, a surgical site of a patient, the robotic arm having a plurality of joints and a plurality of body parts, controlling the robotic arm to perform a surgical procedure at the surgical site of the patient based on the camera tracked surgical site of the patient, and wherein the tracking comprises controlling the movement of the plurality of joints and body parts of the robotic arm to maintain a line of sight of the at least one camera directed towards the surgical site of the patient.

Claims (117)

1. A surgical method comprising:

tracking, via at least one camera attached to a robotic arm of a surgical robot, a surgical site of a patient, the robotic arm having a plurality of joints and a plurality of body parts;

controlling the robotic arm to perform a surgical procedure at the surgical site of the patient based on the camera tracked surgical site;

wherein the tracking comprises controlling movement of the plurality of joints and body parts of the robotic arm to maintain a line of sight of the at least one camera directed towards the surgical site of the patient;

the at least one camera is mounted with a movable actuator to a portion of the robotic arm of the surgical robot; and

the tracking comprises controlling the movement of the plurality of joints and body parts and controlling the movement of the movable actuator to maintain the line of sight of the at least one camera directed towards the surgical site of the patient with the at least one camera and movable actuator spaced from the patient and surgical site during the surgical procedure.

2. The surgical method of claim 1 , wherein:

the controlling comprises controlling a tool attached to an end effector of the robotic arm of the surgical robot to perform the surgical procedure at the surgical site based on the camera tracked surgical site of the patient and a cut plan.

3. The surgical method of claim 1 , further comprising:

modifying a cut plan based on the camera tracked surgical site of the patient; and wherein:

the controlling comprises controlling a tool attached to an end effector of the robotic arm of the surgical robot perform the surgical procedure at the surgical site of the patient based the modified cut plan.

4. The surgical method of claim 1 , wherein:

the tracking comprises tracking, via a first camera attached to the robotic arm of the surgical robot and a second camera attached to the robotic arm of the surgical robot, the surgical site of the patient; and

the controlling comprises controlling the surgical robot to perform the surgical procedure at the surgical site of the patient based on a first camera tracked surgical site of the patient and a second camera tracked surgical site of the patient.

5. The surgical method of claim 1 , wherein:

the tracking comprises tracking, via a pair of cameras attached to an end portion of the robotic arm of the surgical robot, the surgical site of the patient, and wherein a first of the pair of cameras comprises a first field of view of the surgical site of the patient, and a second of the pair of cameras comprises a second field of view of the surgical site of the patient, and the first field of view is different from said second field of view of the surgical site of the patient; and

the controlling comprises controlling the surgical robot to perform the surgical procedure at the surgical site of the patient based on the camera tracked surgical site of the patient based on a first camera tracked surgical site of the patient and a second camera tracked surgical site of the patient.

6. The surgical method of claim 1 , wherein:

the tracking comprises tracking, via the at least one camera attached to the robotic arm of the surgical robot, an object in the surgical site of the patient; and

the controlling comprises controlling the surgical robot to perform the surgical procedure at the surgical site of the patient based on the camera tracked object in the surgical site of the patient.

7. The surgical method of claim 6 , further comprising:

determining comprises determining the position and the orientation of the object based on the camera tracked object; and

the controlling comprises controlling the surgical robot to perform the surgical procedure at the surgical site of the patient based on the determined position and orientation of the camera tracked object in the surgical site of the patient.

8. The surgical method of claim 7 wherein:

the determining comprises determining a position and an orientation of the object based on the camera tracked object and registration to three-dimensional data of the object, and

the controlling comprises controlling the surgical robot to perform the surgical procedure at the surgical site of the patient based on the determined position and orientation of the camera tracked object registered to the three-dimensional data of the object in the surgical site of the patient.

9. The surgical method of claim 7 , wherein:

the determining comprises continuously determining the position and the orientation of the object during the surgical procedure based on the camera tracked object and registration to three-dimensional data of the object, and wherein:

the controlling comprises controlling the surgical robot to perform the surgical procedure at the surgical site of the patient based on the continuously determined position and orientation of the camera tracked object in the surgical site of the patient during the surgical procedure.

10. The surgical method of claim 6 , wherein:

the object comprises a bone of the patient, and the controlling comprises controlling the surgical robot to perform a resection or an excavation of the bone of the patient; or

the object comprises a retractor, and the controlling comprises controlling the surgical robot to perform the surgical procedure and avoid contact with the retractor.

11. The surgical method of claim 1 , wherein:

the tracking comprises tracking, via the at least one camera attached to the robotic arm of the surgical robot, at least one marker attached to an object in the surgical site of the patient; and

the controlling comprises controlling the surgical robot to perform the surgical procedure at the surgical site of the patient based on the camera tracked marker attached to the object in the surgical site of the patient.

12. The surgical method of claim 11 , further comprising:

determining a position and an orientation of the object based on the camera tracked marker attached to the object, and wherein:

the controlling comprises controlling the surgical robot to perform the surgical procedure at the surgical site of the patient based on the determined position and orientation of the object in the surgical site of the patient.

13. The surgical method of claim 11 , further comprising:

determining a position and an orientation of the object based on the camera tracked marker and registration to three-dimensional data of the object, and wherein:

the controlling comprises controlling the surgical robot to perform the surgical procedure at the surgical site of the patient based on the determined position and orientation of the camera tracked object marker registered to the three-dimensional data of the object in the surgical site of the patient.

14. The surgical method of claim 11 , wherein:

the object comprises a bone of the patient, and the controlling comprises controlling the surgical robot to perform a resection or an excavation of the bone of the patient; or

the object comprises a retractor, and the controlling comprises controlling the surgical robot to perform the surgical procedure and avoid contact with the retractor.

15. The surgical method of claim 11 , wherein the at least one camera comprises at least one infrared camera, and the at least one marker comprises and an active LED marker.

16. The surgical method of claim 1 , wherein:

the tracking comprises tracking, via the at least one camera attached to the robotic arm of the surgical robot, an object in the surgical site of the patient; and further comprising:

determining a position and an orientation of the object based on the camera tracked object;

generating a modified cut plan based on a predetermined cut plan and the determined position and orientation of the camera tracked object in the surgical site of the patient; and

the controlling comprises controlling a tool attached to an end effector of the robotic arm of the surgical robot to perform the surgical procedure at the surgical site based a modified cut plan.

17. The surgical method of claim 16 , wherein:

the tracking comprises tracking, via the at least one camera attached to the robotic arm of the surgical robot, at least one marker attached to the object in the surgical site of the patient;

the determining comprises determining a position and an orientation of the object based on the camera tracked marker; and

the controlling comprises controlling the surgical robot to perform the surgical procedure at the surgical site of the patient based on the determined position and orientation of the camera tracked marker attached to the object in the surgical site of the patient.

18. The surgical method of claim 1 , wherein the at least one camera is operably attached to:

a flange adjacent to the end effector of the robotic arm; or

a joint of the robotic arm; or

a base of the surgical robot.

19. A surgical robotic system comprising:

a robot comprising a robotic arm having an end effector, the robotic arm having a plurality of joints and plurality of body parts;

at least one camera operably attached to said robotic arm;

a controller comprising a memory, one or more processors in communication with the memory, and program instructions executable by the one or more processors via the memory to perform a method, the method comprising:

tracking, via at least one camera attached to the robotic arm of the surgical robot;

controlling the robotic arm to perform a surgical procedure at the surgical site of the patient based on the camera tracked surgical site of the patient; and

wherein the tracking comprises controlling movement of the plurality of joints and body parts of the robotic arm to maintain a line of sight of the at least one camera directed towards the surgical site of the patient; and

the at least one camera is mounted with a movable actuator to a portion of the robotic arm of the surgical robot; and

the tracking comprises controlling the movement of the plurality of joints and body parts and controlling the movement of the movable actuator to maintain a line of sight of the at least one camera directed towards the surgical site of the patient with the at least one camera and movable actuator spaced from the patient and surgical site during the surgical procedure.

20. The surgical robotic system of claim 19 , wherein said at least one camera comprises a plurality of cameras attached to the robotic arm of the surgical robot.

21. The surgical robotic system of claim 20 , wherein said plurality of cameras is attached to a flange of the robotic arm of the surgical robot, a first of the cameras comprises a first field of view of the surgical site of the patient, and a second of the cameras comprises a second field of view of the surgical site, and the first field of view is different from said second field of view of the surgical site.

22. The surgical robotic system of claim 19 , wherein:

the at least one camera comprises at least one RGB-D camera; or

the at least one camera comprises at least one infrared camera.

23. The surgical robotic system of claim 19 , wherein the at least one camera is attached to:

a flange adjacent to the end effector of the robotic arm; or

a joint of the robotic arm; or

a base of the surgical robot.

24. The surgical robotic system of claim 19 , wherein the at least one camera comprises a first camera attached to a flange adjacent to the end effector of the robotic arm, a second camera is attached to a joint of the robotic arm, and a third camera attached to a base of the surgical robot.

25. The surgical robotic system of claim 19 , wherein the method further comprises:

the controlling comprising controlling a tool attached to an end effector of the robotic arm of the surgical robot to perform the surgical procedure at the surgical site based on the camera tracked surgical site of the patient and a predetermined cut plan.

26. The surgical robotic system of claim 19 , wherein the method further comprises:

generating a modified cut plan based on a predetermined cut plan and the camera tracked surgical site of the patient; and wherein:

the controlling comprises controlling a tool attached to an end effector of the robotic arm of the surgical robot to perform the surgical procedure at the surgical site based the modified cut plan.

27. The surgical robotic system of claim 19 , wherein the method further comprises:

the tracking comprises tracking, via the at least one camera attached to the robotic arm of the surgical robot, an object in the surgical site of the patient;

the controlling comprises controlling the surgical robot to perform the surgical procedure at the surgical site of the patient based on the camera tracked object in the surgical site of the patient.

28. The surgical robotic system of claim 27 , wherein:

the object comprises a bone and the controlling comprises controlling the surgical robot to perform a resection or an excavation of the bone of the patient; or

the object comprises a retractor, and the controlling comprises controlling the surgical robot to perform the surgical procedure and avoid contact with the retractor.

29. The surgical robotic system of claim 19 , wherein the method further comprises:

determining a position and an orientation of an object based on the camera tracked marker attached to the object, and wherein:

the controlling comprises controlling the surgical robot to perform the surgical procedure at the surgical site of the patient based on the determined position and orientation of the camera tracked object in the surgical site of the patient.

30. The surgical robotic system of claim 19 , wherein the method comprises:

the tracking comprises tracking, via the at least one camera attached to the robotic arm of the surgical robot, an object in the surgical site of the patient; and further comprising:

determining a position and an orientation of the object based on the camera tracked object;

generating a modified cut plan based on a predetermined cut plan and the determined position and orientation of the camera tracked object in the surgical site of the patient; and wherein:

the controlling comprises controlling a tool attached to an end effector of the robotic arm of the surgical robot to perform the surgical procedure at the surgical site based on a modified cut plan.

31. The surgical robotic system of claim 19 , wherein:

the tracking comprises tracking, via the at least one camera attached to the robotic arm of the surgical robot, at least one marker attached to the object in the surgical site of the patient;

the determining comprises determining a position and an orientation of the object based on the camera tracked marker; and

the controlling comprises controlling the surgical robot to perform the surgical procedure at the surgical site of the patient based on the determined position and orientation of the camera tracked marker attached to the object in the surgical site of the patient.

32. The surgical robotic system of claim 19 , wherein the at least one camera is movably mounted relative to the robotic arm of the surgical robot, and wherein:

the tracking comprises tracking, via moving the at least one camera differently from movement of the robotic arm of the surgical robot, the surgical site of the patient;

the controlling comprises controlling the surgical robot to perform a surgical procedure at the surgical site of the patient based on the moving camera tracked surgical site of the patient.

33. A computer program product comprising a non-transitory computer readable storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method, the method comprising:

tracking, via at least one camera attached to a robotic arm of a surgical robot, a surgical site of a patient;

controlling the surgical robot to perform a surgical procedure at the surgical site of the patient based on the camera tracked surgical site of the patient; and wherein:

the at least one camera is mounted with a movable actuator to a portion of the robotic arm of the surgical robot; and

the tracking comprises controlling the movement of the plurality of joints and body parts and controlling the movement of the movable actuator to maintain a line of sight of the at least one camera directed towards the surgical site of the patient with the at least one camera and movable actuator spaced from the patient and surgical site during the surgical procedure.

34. The computer program product of claim 33 , wherein the method comprises:

the tracking comprises tracking, via the at least one camera attached to the robotic arm of the surgical robot, an object in the surgical site of the patient; and further comprising:

determining a position and an orientation of the object based on the camera tracked object;

generating a modified cut plan based on a predetermined cut plan and the determined position and orientation of the camera tracked object in the surgical site of the patient; and wherein:

the controlling comprises controlling a tool attached to an end effector of the robotic arm of the surgical robot to perform the surgical procedure at the surgical site based on the modified cut plan.

35. The computer program product of claim 33 , wherein the at least one camera is movable mounted relative to the robotic arm of the surgical robot, and wherein:

the tracking comprises tracking, via moving the at least one camera differently from the movement of the robotic arm of the surgical robot, the surgical site of the patient;

the controlling comprises controlling the surgical robot to perform a surgical procedure at the surgical site of the patient based on the moving camera tracked surgical site of the patient; and

wherein the tracking comprises controlling the movement of the plurality of joints and body parts of the robotic arm to maintain a line of sight of the at least one camera directed towards the surgical site of the patient.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 3, 2024
From: SEXSON, BENJAMIN; DICICCO, MATTHEW; JIN, BRIAN
To: MONOGRAM ORTHOPAEDICS INC.
Reel/Frame 067903/0662 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 28, 2023
From: UNIS, DOUGLAS B.
To: ICAHN SCHOOL OF MEDICINE AT MOUNT SINAI
Reel/Frame 063480/0168 →
Continuity (5)
Continuation PCTUS2020035408 · May 29, 2020
Continuation PCTUS2020033810 · May 20, 2020
Provisional Application 62854648 · May 30, 2019
Provisional Application 62850050 · May 20, 2019
Related Publication 20220079687A1 · Mar 17, 2022
References Cited (53)
US 9119655B2 · Bowling et al. · 2015 [cited by applicant]
US 9119665B2 · Ahern et al. · 2015 [cited by applicant]
US 9439556B2 · Pandya et al. · 2016 [cited by applicant]
US 9603665B2 · Bowling et al. · 2017 [cited by applicant]
US 9668768B2 · Piron · 2017 [cited by examiner]
US 9788903B2 · Kim et al. · 2017 [cited by applicant]
US 10512509B2 · Bowling et al. · 2019 [cited by applicant]
US 20040106916A1 · Quaid · 2004 [cited by examiner]
US 20060184275A1 · Hosokawa et al. · 2006 [cited by applicant]
US 20110270443A1 · Kamiya et al. · 2011 [cited by applicant]
US 20130010081A1 · Tenney · 2013 [cited by examiner]
US 20130331644A1 · Pandya et al. · 2013 [cited by applicant]
US 20140107471A1 · Haider · 2014 [cited by examiner]
US 20140200621A1 · Malackowski · 2014 [cited by examiner]
US 20140275760A1 · Lee et al. · 2014 [cited by applicant]
US 20140316434A1 · Simaan et al. · 2014 [cited by applicant]
US 20160058521A1 · Liang et al. · 2016 [cited by applicant]
US 20160113728A1 · Piron · 2016 [cited by examiner]
US 20160338782A1 · Bowling et al. · 2016 [cited by applicant]
US 20170056116A1 · Kostrzewski · 2017 [cited by applicant]
US 20170071692A1 · Taylor et al. · 2017 [cited by applicant]
US 20170258526A1 · Lang · 2017 [cited by examiner]
US 20170265943A1 · Sela · 2017 [cited by examiner]
US 20170312031A1 · Amanatullah · 2017 [cited by examiner]
US 20170333137A1 · Roessler · 2017 [cited by examiner]
US 20180000546A1 · Crawford et al. · 2018 [cited by applicant]
US 20180297206A1 · Larkin · 2018 [cited by examiner]
US 20190090965A1 · Farritor · 2019 [cited by examiner]
US 20190142518A1 · Viscardi et al. · 2019 [cited by applicant]
US 20190142520A1 · VanDyken · 2019 [cited by applicant]
US 20190290370A1 · Brummund · 2019 [cited by examiner]
US 20190314097A1 · Diolaiti · 2019 [cited by examiner]
US 20190365481A1 · Otto · 2019 [cited by examiner]
US 20210137634A1 · Lang · 2021 [cited by examiner]
US 20220079687A1 · Sexson · 2022 [cited by examiner]
US 20220125526A1 · Wald · 2022 [cited by examiner]
US 20230126611A1 · Itotani · 2023 [cited by examiner]
US 20230157525A1 · Hunter · 2023 [cited by examiner]
US 20230415353A1 · Oba · 2023 [cited by examiner]
DE 102013211698A1 · 2014 [cited by examiner]
WO 2020236937A1 · 2020 [cited by applicant]
WO 2020243631A1 · 2020 [cited by applicant]
DE-102013211698-A1 translation (Year: 2014). [cited by examiner]
Partial Supplementary European Search Report for European Application No. 20813577.2 dated Jun. 27, 2023, 18 pages. [cited by applicant]
Partial Supplementary European Search Report for European Application No. 20809508.3 dated Jun. 15, 2023, 13 pages. [cited by applicant]
International Preliminary Report on Patentability for International Application No. PCT/US2020/035408, dated Nov. 16, 2021, 6 pages, International Bureau of WIPO. [cited by applicant]
Australian Patent Office, First Examination Report dated Jan. 10, 2022, 4 pages. [cited by applicant]
Sexson et al., International Search Report and Written Opinion for PCT/US2020/035408 (published as WO 2020/243631), titled “Robot Mounted Camera Registration and Tracking System for Orthopedic and Neurological Surgery”,… [cited by applicant]
Sexson et al., International Search Report and Written Opinion for PCT/US2020/033810 (published as WO 2020/236937), titled “A System and Method for Interaction and Definition of Tool Pathways for a Robotic Cutting Tool”… [cited by applicant]
Wang et al., “Robot-assisted Occlusion Avoidance for Surgical Instrument Optical Tracking System”, Proceeding of the 2015 IEEE International Conference on Information and Automation, 6 pages, Aug. 2015. [cited by applicant]
Sexson et al., U.S. Appl. No. 17/455,822, titled “A System and Method for Interaction and Definition of Tool Pathways for a Robotic Cutting Tool”, 57 pages, filed Nov. 19, 2021. [cited by applicant]
Sexson et al., Examination Report, Australian Parent Application No. 2023200152, mailed Feb. 15, 2024, 5 pages, Feb. 15, 2024. [cited by applicant]
Sexson et al., Examination Report, European Parent Application No. 20813577.2, mailed Jun. 20, 2025, 6 pages, Jun. 20, 2025. [cited by applicant]
Cited By (1)
US 12,593,962