IP Library › Granted Patent US 12,575,891
Granted Patent B2
US 12,575,891 · App. 17/533,021 · Granted Mar 17, 2026

AI-based automatic tool presence and workflow/phase/activity recognition

Inventors: Hedyeh Rafii-Tari (Mountain View, CA); Menglong Ye (Mountain View, CA); Mingyi Zheng (San Jose, CA)
Assignee: Auris Health, Inc.
A61B34/20A61B17/29A61B17/3403A61B90/90G06T7/0012G06T7/20A61B2017/2901A61B2034/2065A61B90/92G06T2207/10016G06T2207/20081G06T2207/30061
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,575,891
App. No.
17/533,021
Granted
Mar 17, 2026
Kind
B2
Abstract

A robotic system is configured to automatically identify surgical instruments used during a bronchoscopy procedure. The robotic system can include a video capture device, a robotic manipulator, sensors configured to detect a configuration of the robotic manipulator, and control circuitry communicatively coupled to the robotic manipulator. The control circuitry is configured to perform, using a machine learning classifier, a first analysis of a bronchoscopy video of a patient site to track a medical instrument in the bronchoscopy video. The control circuitry can then identify a set of possible instrument identifications for the medical instrument in the bronchoscopy video based on the first analysis and an identified phase of the bronchoscopy procedure. The control circuitry can then track a motion of the medical instrument in the bronchoscopy video and select an identification from the set of possible instrument identification for the medical instrument based at least on the tracked motion.

Claims (37)

1 . A robotic system for automatically identifying surgical instruments used during a bronchoscopy procedure, the robotic system comprising:

a video capture device;

a robotic manipulator;

one or more sensors configured to detect a configuration of the robotic manipulator;

control circuitry communicatively coupled to the robotic manipulator, the control circuitry configured to:

perform, using a machine learning classifier, a first analysis of a bronchoscopy video of a patient site captured by the video capture device, the first analysis configured to categorize a medical instrument in the bronchoscopy video into a first tool class of a plurality of tool classes;

identify a set of possible instrument types for the medical instrument in the bronchoscopy video based on the first tool class and an identified phase of the bronchoscopy procedure, wherein the set of possible instrument types includes one or more phase-specific medical instruments that are associated with the first tool class and known to be employed for the identified phase;

track a motion of the medical instrument in the bronchoscopy video; and

select a type of instrument from the set of possible instrument types for the medical instrument based at least in part on the tracked motion.

2 . The robotic system of claim 1 , wherein the tracked motion comprises a rotation of the medical instrument and the type of instrument is a radial probe endobronchial ultrasound (REBUS) instrument.

3 . The robotic system of claim 1 , wherein the tracked motion comprises a pull motion of the medical instrument and the type of instrument is a forceps.

4 . The robotic system of claim 1 , wherein the tracked motion comprises a back and forth dithering motion of the medical instrument and the type of instrument is a needle or a brush.

5 . The robotic system of claim 4 , wherein a color of the medical instrument is used to further identify the medical instrument as the needle.

6 . The robotic system of claim 1 , wherein the plurality of tool classes comprises at least a sheath class, a needle class, a REBUS class, a forceps class, a brush class, and a background class.

7 . The robotic system of claim 1 , wherein the plurality of tool classes comprises at least a sheath class and a needle class.

8 . The robotic system of claim 7 , wherein the sheath class comprises medical instruments including a REBUS, a needle, and a forceps.

9 . The robotic system of claim 7 , wherein the needle class comprises medical instruments including a needle tip, a brush, or a sheath.

10 . The robotic system of claim 1 , wherein the control circuitry is further configured to identify a phase of the bronchoscopy procedure based at least in part on sensor data from the one or more sensors.

11 . A method for automatically identifying surgical instruments used during a bronchoscopy procedure by a robotic system, the method comprising:

performing, using a machine learning classifier, a first analysis of a bronchoscopy video of a patient site captured by a video capture device of the robotic system, the first analysis configured to categorize a medical instrument in the bronchoscopy video into a first tool class of a plurality of tool classes;

identifying a set of possible instrument types for the medical instrument in the bronchoscopy video based on the first tool class and an identified phase of the bronchoscopy procedure, wherein the set of possible instrument types includes one or more phase-specific medical instruments associated with the first tool class and known to be employed for the identified phase;

tracking a motion of the medical instrument in the bronchoscopy video; and

selecting a type of instrument from the set of possible instrument types for the medical instrument based at least in part on the tracked motion.

12 . The method of claim 11 , wherein the tracked motion comprises a rotation of the medical instrument and the type of instrument is a radial probe endobronchial ultrasound (REBUS) instrument.

13 . The method of claim 11 , wherein the tracked motion comprises a pull motion of the medical instrument and the type of instrument is a forceps.

14 . The method of claim 11 , wherein the tracked motion comprises a back and forth dithering motion of the medical instrument and the type of instrument is a needle or a brush.

15 . The method of claim 11 , further comprising:

identifying the phase of the bronchoscopy procedure based at least in part on sensor data from the robotic system.

16 . The method of claim 11 , wherein the plurality of classes comprise at least a REBUS class, a forceps class, a brush class, and a background class.

17 . The method of claim 11 , wherein the machine learning classifier is further configured to perform an episode recognition process.

18 . The method of claim 17 , wherein the episode recognition process comprises classifying the medical instrument into one of several classes based at least in part on the tracked motion.

19 . A computerized system for automatically identifying surgical instruments used during a bronchoscopy procedure, the computerized system comprising:

one or more processors configured to:

perform, using a machine learning classifier, a first analysis of a bronchoscopy video of a patient site captured by a video capture device, the first analysis configured to categorize a medical instrument in the bronchoscopy video into a first tool class of a plurality of tool classes;

identify a set of possible instrument types for the medical instrument in the bronchoscopy video based on the first tool class and an identified phase of the bronchoscopy procedure, wherein the set of possible instrument types includes one or more phase-specific medical instruments associated with the first tool class and known to be employed for the identified phase;

track a motion of the medical instrument in the bronchoscopy video; and

select a type of instrument from the set of possible instrument types for the medical instrument based at least in part on the tracked motion.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 10, 2022
From: RAFII-TARI, HEDYEH; YE, MENGLONG; ZHENG, MINGYI
To: AURIS HEALTH, INC.
Reel/Frame 058595/0027 →
Continuity (4)
Provisional Application 63244224 · Sep 14, 2021
Provisional Application 63156251 · Mar 3, 2021
Provisional Application 63116768 · Nov 20, 2020
Related Publication 20220160433A1 · May 26, 2022
References Cited (46)
US 12093813B2 · Yao · 2024 [cited by examiner]
US 20200226751A1 · Jin · 2020 [cited by examiner]
US 20200273575A1 · Wolf · 2020 [cited by examiner]
US 20200297444A1 · Camarillo · 2020 [cited by examiner]
US 20210282892A1 · Kim · 2021 [cited by examiner]
US 20220040855A1 · Cha · 2022 [cited by examiner]
US 20230093555A1 · Ayvali · 2023 [cited by examiner]
US 20240203567A1 · Ruiz · 2024 [cited by examiner]
WO WO2017205373A1 · 2017 [cited by examiner]
Namazi, Babak, Ganesh Sankaranarayanan, and Venkat Devarajan. “LapTool-Net: a contextual detector of surgical tools in laparoscopic videos based on recurrent convolutional neural networks.” arXiv preprint arXiv:1905.089… [cited by examiner]
A. Jin et al., “Tool Detection and Operative Skill Assessment in Surgical Videos Using Region-Based Convolutional Neural Networks,” 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, … [cited by examiner]
Namazi, B., Sankaranarayanan, G. & Devarajan, V. A contextual detector of surgical tools in laparoscopic videos using deep learning. Surg Endosc 36, 679-688 (2022). https://doi.org/10.1007/s00464-021-08336-x (Year: 2022… [cited by examiner]
0. Zisimopoulos, E. Flouty, I. Luengo, P. Giataganas, J. Nehme, A. Chow, and D. Stoyanov, “Deepphase: surgical phase recognition in cataracts videos,” in International Conference on Medical Image Computing and Computer-… [cited by applicant]
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXi… [cited by applicant]
A. P. Twinanda, “Vision-based approaches for surgical activity recognition using laparoscopic and rbgd videos,” Ph.D. dissertation, Strasbourg, 2017. [cited by applicant]
A. P. Twinanda, S. Shehata, D. Mutter, J. Marescaux, M. De Mathelin, and N. Padoy, “Endonet: a deep architecture for recognition tasks on laparoscopic videos,” IEEE transactions on medical imaging, vol. 36, No. I, 2016,… [cited by applicant]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, Attention is all you need; in Advances in neural information processing systems, 2017, 15 pages. [cited by applicant]
A. Zia, A. Hung, I. Essa, and A. Jarc, “Surgical activity recognition in robot-assisted radical prostatectomy using deep learning,” in International Conference on Medical Image Computing and ComputerAssisted Interventio… [cited by applicant]
C. I. Nwoye, T. Yu, C. Gonzalez, B. Seeliger, P. Mascagni, D. Mutter, J. Marescaux, and N. Padoy, “Rendezvous: Attention mechanisms for the recognition of surgical action triplets in endoscopic videos,” arXiv preprint a… [cited by applicant]
C. Lea, R. Vidal, A. Reiter, and G.D. Hager, “Temporal convolutional networks: A unified approach to action segmentation,” in European Conference on Computer Vision. Springer, 2016, 4 pages. [cited by applicant]
D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),” arXiv preprint arXiv:1606.08415, 2016, 9 pages. [cited by applicant]
D. Rivoir, S. Bodenstedt, I. Funke, F. von Bechtolsheim, M. Distler, J. Weitz, and S. Speidel, “Rethinking anticipation tasks: Uncertainty aware anticipation of sparse surgical instrument usage for context aware assista… [cited by applicant]
F. Lalys, D. Bouget, L. Riffaud, and P. Jannin, “Automatic knowledge-based recognition of low-level tasks in ophthalmological procedures,” International journal of computer assisted radiology and surgery, vol. 8, No. 1,… [cited by applicant]
G. Lecuyer, M. Ragot, N. Martin, L. Launay, and P. Jannin, “Assisted phase and step annotation for surgical videos,” International journal of computer assisted radiology and surgery, 2020, 18 pages. [cited by applicant]
G. Yengera, D. Mutter, J. Marescaux, and N. Padoy, “Less is more: Surgical phase recognition with less annotations through self-supervised pre-training of cnn-lstrn networks,” arXiv preprint arXiv:1805.08569, 2018, 15 p… [cited by applicant]
I. Funke, S. Bodenstedt, F. Oehme, F. von Bechtolsheim, J. Weitz, and S. Speidel, “Using 3d convolutional neural networks to learn spatiotemporal features for automatic surgical gesture recognition in video,” in Interna… [cited by applicant]
I. Funke, S. T. Mees, J. Weitz, and S. Speidel, “Video-based surgical skill assessment using 3d convolutional neural networks,” International journal of computer assisted radiology and surgery, vol. 14, No. 7, 2019, 15 … [cited by applicant]
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, 9 pages. [cited by applicant]
K. He, X. Zhang, S. Ren, J. Sun, Identity mappings in deep residual networks; 2016, 16 pages. [cited by applicant]
L. A. Torre, R. L. Siegel, and A. Jemal, “Lung cancer statistics,” Lung cancer and personalized medicine, 2016, 19 pages. [cited by applicant]
L. Ding and C. Xu, “Tricomet: A hybrid temporal convolutional and recurrent network for video action segmentation,” arXiv preprint arXiv:1705.07818, 2017, 10 pages. [cited by applicant]
L. Maier-Hein, S. S. Vedula, S. Speidel, N. Navab, R. Kikinis, A. Park, M. Eisenmann, H. Feussner, G. Forestier, S. Giannarou et al., “Surgical data science for next-generation interventions,” Nature Biomedical Engineer… [cited by applicant]
M. Mohamed, G. Cesa, T. S. Cohen, and M. Welling, “A data and compute efficient design for limited-resources deep learning,” ar Xiv preprint arXiv:2004.09691, 2020, 8 pages. [cited by applicant]
N. Padoy, Machine and deep learning for workflow recognition during surgery, Minimally Invasive Therapy & Allied Technologies, vol. 28, No. 2, 2019, 10 pages. [cited by applicant]
N. Padoy, T. Blum, S.-A. Ahmadi, H. Feussner, M.-0. Berger, and N. Navab, “Statistical modeling and recognition of surgical workflow,” Medical image analysis, vol. 16, 2012, 22 pages. [cited by applicant]
P. Lei and S. Todorovic, Temporal deformable residual networks for action segmentation in videos; in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, 10 pages. [cited by applicant]
S. D. Murgu, “Robotic assisted-bronchoscopy: technical tips and lessons learned from the initial experience with sampling peripheral lung lesions,” BMC pulmonary medicine, vol. 19, No. I, 2019, 8 pages. [cited by applicant]
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, No. 8, 1997, 31 pages. [cited by applicant]
T. Czempiel, M. Paschali, M. Keicher, W. Simson, H. Feussner, S. T. Kim, and N. Navab, “Tecno: Surgical phase recognition with multistage temporal convolutional networks,” in International Conference on Medical Image Co… [cited by applicant]
X. Gao, Y. Jin, Y. Long, Q. Dou, and P.-A. Heng, “Trans-svnet: Accurate phase recognition from surgical videos via hybrid embedding aggregation transformer,” arXiv preprint arXiv:2103.09712, 2021, 13 pages. [cited by applicant]
Y. A. Farha and J. Gall, “Ms-ten: Multi-stage temporal convolutional network for action segmentation,” in Proceedings of the IEEEICVF Conference on Computer Vision and Pattern Recognition, 2019, 10 pages. [cited by applicant]
Y. Jin, H. Li, Q. Dou, H. Chen, J. Qin, C.-W. Fu, and P.-A. Heng, “Multi-task recurrent convolutional network with correlation loss for surgical video analysis,” Medical image analysis, vol. 59, 2020, 31 pages. [cited by applicant]
Y. Jin, Q. Dou, H. Chen, L. Yu, J. Qin, C.-W. Fu, and P.-A. Heng, “Svrcnet: workflow recognition from surgical videos using recurrent convolutional network,” IEEE transactions on medical imaging, vol. 37, No. 5, 2017, 1… [cited by applicant]
Y. Qin, S. A. Pedram, S. Feyzabadi, M. Allan, A. J. McLeod, J. W. Burdick, and M. Azizian, “Temporal segmentation of surgical subtasks through deep learning with multiple data sources,” in 2020 IEEE International Confer… [cited by applicant]
Mingyi Zheng, Menglong Ye, Hedyeh Rafii-Tari, Automatic Biopsy Tool Presence and Episode Recognition in Robotic Bronchoscopy Using a Multi-Task Vision Transformer Network, IEEE—International Conference on Robotics and A… [cited by applicant]
Sharghi et al., “Automatic Operating Room Surgical Activity Recognition for Robot-Assisted Surgery,” arXiv:2006.16166v1 [cs.CV] Jun. 29, 2020, 10 pages. [cited by applicant]