IP Library Granted Patent US 12,451,239
Granted Patent B2
US 12,451,239 · App. 18/037,976 · Granted Oct 21, 2025

Systems and methods for assessing surgical ability

Inventors: Kristen Brown (Roswell, GA); Kiran Bhattacharyya (Atlanta, GA); Anthony Michael Jarc (Johns Creek, GA); Sue Kulason (Baltimore, MD); Linlin Zhou (Alpharetta, GA); Aneeq Zia (Atlanta, GA)
Assignee: Intuitive Surgical Operations, Inc.
G16H40/20
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,451,239
App. No.
18/037,976
Granted
Oct 21, 2025
Kind
B2
Abstract

Various of the disclosed embodiments relate to computer systems and computer-implemented methods for measuring and monitoring surgical performance, For example, the system may receive raw data acquired from the surgical theater, generate and select features from the data amenable to analysis, and then train a machine learning classifier using the selected features to facilitate subsequent assessment of other surgeons' performances. Generation and selection of the features may itself involve application of a machine learning classifier in some embodiments. While some embodiments contemplate raw data acquired from surgical robotic systems, some embodiments facilitate assessments upon data acquired from non-robotic surgical theaters.

Claims (67)

1. A computer-implemented method for generating a score based upon surgical data, the method comprising:

determining generating one or more metric values of a plurality of operational performance indicators (OPIs) from based at least upon surgical data;

establishing identifying a data structure comprising a mapping of each of the plurality of OPIs to at least one skill of a plurality of skills;

selecting, from the data structure comprising the mapping, one or more OPIs of the plurality of OPIs based at least on a skill from the plurality of skills;

identifying one or more metric values of the selected one or more OPIs for the user; and

generating a score for the skill of the user at least in part, by providing the one or more metric values for the selected one or more OPIs of the user as input to a machine learning model trained for the skill.

2. The computer-implemented method of claim 1 , wherein generating the score comprises:

providing the one or more metric values based upon the surgical data to the machine learning model implementation to generate a prediction; and

mapping the prediction to the score.

3. The computer-implemented method of claim 2 , wherein the prediction indicates whether the surgical data is more closely associated with a first classification or with a second classification.

4. The computer-implemented method of claim 3 , wherein,

the first classification and the second classification are associated with different surgeon skill levels, and wherein,

the machine learning model implementation is configured to receive metric values corresponding to surgical data associated with a particular surgical task.

5. The computer-implemented method of claim 4 , wherein the surgical data comprises patient-side kinematics data, operator-side kinematics data, visualization tool video data, and system events data.

6. The computer-implemented method of claim 2 , wherein,

determining the one or more metric values based upon the surgical data comprises providing the surgical data to a filtered corpus of metrics determined from annotated surgical data, wherein,

generating the score comprises:

inputting the metric values to the machine learning model implementation; and

applying the output of the machine learning model implementation to a mapping of predictions to scores, and wherein,

the machine learning model implementation is configured to receive metric values corresponding to the filtered corpus of metrics.

7. The computer-implemented method of claim 6 , the method further comprising:

presenting the score as one of several scores generated over the course of at least a portion of a surgeon's surgical performance.

8. A non-transitory computer-readable medium comprising instructions configured to cause a computer system to perform a method for generating a score based upon surgical data, the method comprising:

determining generating one or more metric values of a plurality of operational performance indicators (OPIs) from based at least upon surgical data;

establishing identifying a data structure comprising a mapping of each of the plurality of OPIs to at least one skill of a plurality of skills;

selecting, from the data structure comprising the mapping, one or more OPIs of the plurality of OPIs based at least on a skill from the plurality of skills;

identifying one or more metric values of the selected one or more OPIs for the user; and

generating a score for the skill of the user at least in part, by providing the one or more metric values for the selected one or more OPIs of the user as input to a machine learning model trained for the skill.

9. The non-transitory computer-readable medium of claim 8 , wherein generating the score comprises:

providing the one or more metric values based upon the surgical data to the machine learning model implementation to generate a prediction; and

mapping the prediction to the score.

10. The non-transitory computer-readable medium of claim 9 , wherein the prediction indicates whether the surgical data is more closely associated with a first classification or with a second classification.

11. The non-transitory computer-readable medium of claim 10 , wherein,

the first classification and the second classification are associated with different surgeon skill levels, and wherein,

the machine learning model implementation is configured to receive metric values corresponding to surgical data associated with a particular surgical task.

12. The non-transitory computer-readable medium of claim 11 , wherein the surgical data comprises patient-side kinematics data, operator-side kinematics data, visualization tool video data, and system events data.

13. The non-transitory computer-readable medium of claim 9 , wherein,

determining the one or more metric values based upon the surgical data comprises providing the surgical data to a filtered corpus of metrics determined from annotated surgical data, wherein,

generating the score comprises:

inputting the metric values to the machine learning model implementation; and

applying the output of the machine learning model implementation to a mapping of predictions to scores, and wherein,

the machine learning model implementation is configured to receive metric values corresponding to the filtered corpus of metrics.

14. The non-transitory computer-readable medium of claim 13 , the method further comprising:

presenting the score as one of several scores generated over the course of at least a portion of a surgeon's surgical performance.

15. A computer system comprising:

at least one processor;

at least one memory comprising instructions configured to cause the computer system to perform a method for generating a score based upon surgical data, the method comprising:

determining generating one or more metric values of a plurality of operational performance indicators (OPIs) from based at least upon surgical data;

establishing identifying a data structure comprising a mapping of each of the plurality of OPIs to at least one skill of a plurality of skills;

selecting, from the data structure comprising the mapping, one or more OPIs of the plurality of OPIs based at least on a skill from the plurality of skills;

identifying one or more metric values of the selected one or more OPIs for the user; and

generating a score for the skill of the user at least in part, by providing the one or more metric values for the selected one or more OPIs of the user as input to a machine learning model trained for the skill.

16. The computer system of claim 15 , wherein generating the score comprises:

providing the one or more metric values based upon the surgical data to the machine learning model implementation to generate a prediction; and

mapping the prediction to the score.

17. The computer system of claim 16 , wherein the prediction indicates whether the surgical data is more closely associated with a first classification or with a second classification.

18. The computer system of claim 17 wherein,

the first classification and the second classification are associated with different surgeon skill levels, and wherein,

the machine learning model implementation is configured to receive metric values corresponding to surgical data associated with a particular surgical task.

19. The computer system of claim 16 , wherein,

determining the one or more metric values based upon the surgical data comprises providing the surgical data to a filtered corpus of metrics determined from annotated surgical data, wherein,

generating the score comprises:

inputting the metric values to the machine learning model implementation; and

applying the output of the machine learning model implementation to a mapping of predictions to scores, and wherein,

the machine learning model implementation is configured to receive metric values corresponding to the filtered corpus of metrics.

20. The computer system of claim 19 , the method further comprising:

presenting the score as one of several scores generated over the course of at least a portion of a surgeon's surgical performance.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 19, 2023
From: BROWN, KRISTEN; BHATTACHARYYA, KIRAN; JARC, ANTHONY MICHAEL; KULASON, SUE; ZHOU, LINLIN; ZIA, ANEEQ
To: INTUITIVE SURGICAL OPERATIONS, INC.
Reel/Frame 063708/0642 →
Continuity (2)
Provisional Application 63121220 · Dec 3, 2020
Related Publication 20230317258A1 · Oct 5, 2023
References Cited (192)
US 9283675B2 · Hager et al. · 2016 [cited by applicant]
US 9472121B2 · Pravong · 2016 [cited by examiner]
US 9548002B2 · Black · 2017 [cited by examiner]
US 9788907B1 · Alvi · 2017 [cited by examiner]
US 9836654B1 · Alvi · 2017 [cited by examiner]
US 9847044B1 · Foster · 2017 [cited by examiner]
US 9852053B2 · Verkoeyen · 2017 [cited by examiner]
US 9898937B2 · Poulsen · 2018 [cited by examiner]
US 9922172B1 · Alvi · 2018 [cited by examiner]
US 9922579B2 · Black · 2018 [cited by examiner]
US 9940849B2 · Hart · 2018 [cited by examiner]
US 9959786B2 · Breslin · 2018 [cited by examiner]
US 10026337B2 · Black · 2018 [cited by examiner]
US 10081727B2 · Felsinger · 2018 [cited by examiner]
US 10121391B2 · Breslin · 2018 [cited by examiner]
US 10140889B2 · Black · 2018 [cited by examiner]
US 10198966B2 · Wachli · 2019 [cited by examiner]
US 10223936B2 · Black · 2019 [cited by examiner]
US 10332425B2 · Hofstetter · 2019 [cited by examiner]
US 10354556B2 · Hofstetter · 2019 [cited by examiner]
US 10395559B2 · Poulsen · 2019 [cited by examiner]
US D866661S · Munro · 2019 [cited by examiner]
US 10467923B2 · Livneh · 2019 [cited by examiner]
US 10490105B2 · Saleh · 2019 [cited by examiner]
US 10535281B2 · Breslin · 2020 [cited by examiner]
US 10559227B2 · Gmeiner · 2020 [cited by examiner]
US 10572734B2 · Alvi · 2020 [cited by examiner]
US 10657845B2 · Black · 2020 [cited by examiner]
US 10679520B2 · Hofstetter · 2020 [cited by examiner]
US 10692395B2 · Mackenzie · 2020 [cited by examiner]
US 10706743B2 · Black · 2020 [cited by examiner]
US 10720084B2 · Black · 2020 [cited by examiner]
US 10729502B1 · Wolf · 2020 [cited by examiner]
US 10733908B2 · Black · 2020 [cited by examiner]
US 10755602B2 · Hofstetter · 2020 [cited by examiner]
US 10779772B2 · Pan · 2020 [cited by examiner]
US 10785540B2 · Stanis · 2020 [cited by examiner]
US 10791301B1 · Garcia Kilroy · 2020 [cited by examiner]
US 10796606B2 · Felsinger · 2020 [cited by examiner]
US 10847057B2 · Felsinger · 2020 [cited by examiner]
US 10854112B2 · Pravong · 2020 [cited by examiner]
US 10886015B2 · Wolf · 2021 [cited by examiner]
US 10902745B2 · Singh · 2021 [cited by examiner]
US 10943682B2 · Wolf · 2021 [cited by examiner]
US 11030922B2 · Velasco · 2021 [cited by examiner]
US 11034831B2 · Felsinger · 2021 [cited by examiner]
US 11049418B2 · Black · 2021 [cited by examiner]
US 11065079B2 · Wolf · 2021 [cited by examiner]
US 11081229B2 · Alvi · 2021 [cited by examiner]
US 11100815B2 · Hofstetter · 2021 [cited by examiner]
US 11116587B2 · Wolf · 2021 [cited by examiner]
US 11120708B2 · Hofstetter · 2021 [cited by examiner]
US 11189195B2 · Munro · 2021 [cited by examiner]
US 11190847B2 · Stanis · 2021 [cited by examiner]
US 11205103B2 · Zhang · 2021 [cited by examiner]
US 11224485B2 · Wolf · 2022 [cited by examiner]
US 11227686B2 · Makrinich · 2022 [cited by examiner]
US 11343464B2 · Sugano · 2022 [cited by examiner]
US 11348235B2 · Barral · 2022 [cited by examiner]
US 11348682B2 · Wolf · 2022 [cited by examiner]
US 11361679B2 · Hofstetter · 2022 [cited by examiner]
US 11380431B2 · Wolf · 2022 [cited by examiner]
US 11403968B2 · Hart · 2022 [cited by examiner]
US 11426255B2 · Wolf · 2022 [cited by examiner]
US 11450236B2 · Wachli · 2022 [cited by examiner]
US 11452576B2 · Wolf · 2022 [cited by examiner]
US 11484384B2 · Wolf · 2022 [cited by examiner]
US 11514819B2 · Breslin · 2022 [cited by examiner]
US 11568762B2 · Munro · 2023 [cited by examiner]
US 11587466B2 · Hofstetter · 2023 [cited by examiner]
US 11605161B2 · Jin · 2023 [cited by examiner]
US 11636945B1 · Gallegos · 2023 [cited by examiner]
US 11763923B2 · Wolf · 2023 [cited by examiner]
US 11769207B2 · Wolf · 2023 [cited by examiner]
US 11798092B2 · Wolf · 2023 [cited by examiner]
US 11836917B2 · Barral · 2023 [cited by examiner]
US 11847936B2 · Jarc · 2023 [cited by examiner]
US 11854425B2 · Black · 2023 [cited by examiner]
US 11869378B2 · Hofstetter · 2024 [cited by examiner]
US 11887504B2 · Hofstetter · 2024 [cited by examiner]
US 11941050B2 · Barral · 2024 [cited by examiner]
US 11971951B2 · Wang · 2024 [cited by examiner]
US 11990055B2 · Breslin · 2024 [cited by examiner]
US 12033104B2 · Wolf · 2024 [cited by examiner]
US 12048414B2 · Sasada · 2024 [cited by examiner]
US 12087179B2 · Hofstetter · 2024 [cited by examiner]
US 12131664B2 · Hofstetter · 2024 [cited by examiner]
US 12137992B2 · Jin · 2024 [cited by examiner]
US 12154454B2 · Pravong · 2024 [cited by examiner]
US 20140220527A1 · Li · 2014 [cited by examiner]
US 20140282199A1 · Basu · 2014 [cited by examiner]
US 20150044654A1 · Lendvay et al. · 2015 [cited by applicant]
US 20150304560A1 · Joshi · 2015 [cited by examiner]
US 20150334075A1 · Wang · 2015 [cited by examiner]
US 20170053543A1 · Agrawal · 2017 [cited by examiner]
US 20170300752A1 · Biswas · 2017 [cited by examiner]
US 20200082299A1 · Vasconcelos · 2020 [cited by examiner]
US 20210315650A1 · Tokarchuk et al. · 2021 [cited by applicant]
US 20210407309A1 · Jarc · 2021 [cited by examiner]
US 20220172119A1 · Welsh · 2022 [cited by examiner]
US 20220270750A1 · Grantcharov et al. · 2022 [cited by applicant]
US 20240071243A1 · Jarc · 2024 [cited by examiner]
US 20240354588A1 · Silva Tavares · 2024 [cited by examiner]
KR 101247513B1 · 2013 [cited by applicant]
WO WO2012060901A1 · 2012 [cited by applicant]
WO WO2020102773A1 · 2020 [cited by applicant]
WO WO2022119754A1 · 2022 [cited by applicant]
WO WO2022231993A1 · 2022 [cited by applicant]
1603.06995v4, Cui, Arxiv, 2016, pp. 1-10. [cited by examiner]
Ambient Intelligence, Chatzigiannakis, Springer, 2019, pp. 330-336. [cited by examiner]
Bouget, Elsevier, 2016, pp. 633-654. [cited by examiner]
Forestier, Elsevier, 2018, pp. 3-11. [cited by examiner]
Hajj, Elsevier, 2018, pp. 203-218. [cited by examiner]
Wang, Springer, 2018, pp. 1959-1970. [cited by examiner]
Abolmaesumi, 2012, Springer, pp. 167-177. [cited by examiner]
Frangi, Springer, 2018, pp. 214-221. [cited by examiner]
Medical Image Computing and Computer Assisted Intervention, Shen, Springer, 2019, pp. 476-484. [cited by examiner]
Pattern Recognition Letters, Danelljan, Elsevier, 2019, pp. 74-81. [cited by examiner]
Towards near real-time assessment of surgical skills, Anh, Springer, 2019, pp. 1-9. [cited by examiner]
Wang, Springer, 2022, pp. 541-550, 2022. [cited by examiner]
Zhu, Nov. 10, 2024, Arxiv, pp. 1-18. [cited by examiner]
Abdi, H., “Bonferroni and Sidak Corrections for Multiple Comparisons,”, indicated as corresponding to “Bonferroni and Sidak corrections for multiple comparisons. In N.J. Salkind (Ed.): Encyclopedia of Measurement and St… [cited by applicant]
Azari, D., et al., “In Search of Characterizing Surgical Skill,” Journal of Surgical Education, 2019, vol. 76 (5), pp. 1348-1363. Retrieved from an account with “Reprints Desk” [URL: https://www.researchsolutions.com/],… [cited by applicant]
Batista, G.E., et al., “A Study of the Behavior of Several Methods for Balancing Machine Learning Training Data,” ACM SIGKDD Explorations Newsletter, Jun. 2004, vol. 6(1), pp. 20-29. Retrieved from an account with “Repr… [cited by applicant]
Berniker, M., et al. “A Probabilistic Approach to Surgical Tasks and Skill Metrics,” Preprint (having a 2017 drafting placeholder date) corresponding to IEEE Transactions on Biomedical Engineering 69.7 (2021): 2212-2219… [cited by applicant]
Birkmeyer, J.D., et al., “Surgical Skill and Complication Rates After Bariatric Surgery,” The New England Journal of Medicine, Oct. 2013, vol. 369(15), pp. 1434-1442. Retrieved from [URL: https://www.nejm.org/doi/pdf/10… [cited by applicant]
Brodersen, K.H., et al., online PDF indicated as corresponding to “The Balanced Accuracy and its Posterior Distribution,” IEEE International Conference on Pattern Recognition, Aug. 2010, pp. 3121-3124. Retrieved from [U… [cited by applicant]
Brown, K., et al., “How to Bring Surgery to the Next Level: Interpretable Skills Assessment in Robotic-Assisted Surgery” as received in connection with the International Search Report and Written Opinion for Application… [cited by applicant]
Caetano-Anolles, D., “Rank Sum Test,” GATK team, webpage [URL: https://gatk.broadinstitute.org/hc/en-us/articles/360035531952-Rank-Sum-Test], as of Nov. 29, 2020. Retrieved from Internet Archive Wayback Machine [URL: ht… [cited by applicant]
Chen, A., et al., “Comparison of Clinical Outcomes and Automated Performance Metrics in Robot-Assisted Radical Prostatectomy With and Without Trainee Involvement,” World Journal of Urology, 2019, vol. 38 (7), pp. 1615-1… [cited by applicant]
Chen, J., et al., “Effect of Surgeon Experience and Bony Pelvic Dimensions on Surgical Performance and Patient Outcomes in Robot-Assisted Radical Prostatectomy,” BJU International, Nov. 2019, vol. 124(5), pp. 828-835. R… [cited by applicant]
Chen, J., et al., “Objective Assessment of Robotic Surgical Technical Skill: A Systematic Review,” The Journal of Urology, Mar. 2019, vol. 201(3), pp. 461-469. Retrieved from [URL: https://www.auajournals.org/dol/pdf/10… [cited by applicant]
Chicco, D. and Jurman, G., “The Advantages of the Matthews Correlation Coefficient (MCC) over F1 Score and Accuracy in Binary Classification Evaluation,” BMC Genomics, Jan. 2020, vol. 21(1), pp. 1-13. Retrieved from [UR… [cited by applicant]
Curry, M., et al., “Objective Assessment in Residency-based Training for Transoral Robotic Surgery,” The Laryngoscope, Oct. 2012, vol. 122 (10), pp. 2184-2192. Retrieved from [URL: https://www.ncbi.nlm.nih.gov/pmc/artic… [cited by applicant]
Dev, H., et al., “Detailed Analysis of Operating Time Learning Curves in Robotic Prostatectomy by a Novice Surgeon,” BJU International, Apr. 2012, vol. 109 (7), pp. 1074-1080. Retrieved from an account with “Reprints De… [cited by applicant]
Dipietro, R., et al., “Recognizing Surgical Activities with Recurrent Neural Networks” arXiv preprint arXiv: 1606.06329 (2016). Retrieved from [URL: https://arxiv.org/pdf/1606.06329.pdf] on Aug. 8, 2023, 08 pages. [cited by applicant]
El-Saig, D., et al., “A Graphical Tool for Parsing and Inspecting Surgical Robotic Datasets” as received in connection with the International Search Report and Written Opinion for Application No. PCT/US2022/026080, mail… [cited by applicant]
Estrada, S., et al., “Smoothness of Surgical Tool Tip Motion Correlates to Skill in Endovascular Tasks,” IEEE Transactions on Human-Machine Systems, Oct. 2016, vol. 46(5), pp. 647-659. Retrieved from [URL: https://mahil… [cited by applicant]
Fard, M.J., et al., online PDF indicated as corresponding to “Automated Robot-Assisted Surgical Skill Evaluation: Predictive Analytics Approach,” The International Journal of Medical Robotics + Computer Assisted Surgery… [cited by applicant]
Fecso, A.B., et al., “Relationship Between Intraoperative Non-technical Performance and Technical Events in Bariatric Surgery,” The British Journal of Surgery, Jul. 2018, vol. 105(8), pp. 1044-1050. Retrieved from [URL:… [cited by applicant]
Gao, Y., et al., “JHU-ISI Gesture and Skill Assessment Working Set ( JIGSAWS ): A Surgical Activity Dataset for Human Motion Modeling,” In MICCAI Workshop: M2CAI, 2014, pp. 1-10. Retrieved from [URL: https://cirl.lcsr.j… [cited by applicant]
Ghani, K.R., et al., “Measuring to Improve: Peer and Crowd-sourced Assessments of Technical Skill with Robot-Assisted Radical Prostatectomy,” European Urology, Apr. 2016, vol. 69(4), pp. 547-550. Retrieved from [URL: ht… [cited by applicant]
Gibaud, B., et al., online PDF indicated as corresponding to “Toward a Standard Ontology of Surgical Process Models.” International Journal of Computer Assisted Radiology and Surgery, Sep. 2018, vol. 13 (9), pp. 1-13. R… [cited by applicant]
Goh, A.C., et al., “Global Evaluative Assessment of Robotic Skills: Validation of a Clinical Assessment Tool to Measure Robotic Surgical Skills,” The Journal of Urology, Jan. 2012, vol. 187(1), pp. 247-252. Retrieved fr… [cited by applicant]
Goldenberg, M.G., et al., “Implementing Assessments of Robot-Assisted Technical Skill in Urological Education: A Systematic Review and Synthesis of the Validity Evidence,” BJU International, Sep. 2018, vol. 122(3), pp. … [cited by applicant]
Goldenberg, M.G., “Evidence That Surgical Performance Predicts Clinical Outcomes,” World Journal of Urology, Jun. 2019, vol. 38 (7), pp. 1-3. Retrieved from an account with “Reprints Desk” [URL: https://www.researchsolu… [cited by applicant]
Grantcharov, T.P. and Reznick, R.K., “Teaching Procedural Skills,” BMJ, May 2008, vol. 336(7653), pp. 1129-1131. Retrieved from [URL: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2386620/pdf/bmj-336-7653-prac-01129.pdf]… [cited by applicant]
Guan, D., et al., “A Review of Ensemble Leaming Based Feature Selection,” IETE Technical Review, Jun. 2014, vol. 31(3), pp. 190-198. Retrieved from an account with “Reprints Desk” [URL: https://www.researchsolutions.com… [cited by applicant]
Hung, A.J., et al., “A Deep-learning Model Using Automated Performance Metrics and Clinical Features to Predict Urinary Continence Recovery After Robot-Assisted Radical Prostatectomy,” BJU International, Sep. 2019, vol.… [cited by applicant]
Hung, A.J., et al., “Development and Validation of Objective Performance Metrics for Robot-Assisted Radical Prostatectomy: a Pilot Study,” The Journal of Urology, Jan. 2018, vol. 199 (1), pp. 296-304. Retrieved from [UR… [cited by applicant]
Hung, A.J., et al., “Experts vs Super-experts: Differences in Automated Performance Metrics and Clinical Outcomes for Robot-Assisted Radical Prostatectomy,” BJU International, May 2019, vol. 123 (5), pp. 861-868. Retrie… [cited by applicant]
Hung, A.J., et al., “Structured Learning for Robotic Surgery Utilizing a Proficiency Score: a Pilot Study,” understood to correspond to World Journal of Urology, Jan. 2017, vol. 35 (1), pp. 27-34 though indicated as pub… [cited by applicant]
Hung, A.J., et al., “Surgeon Automated Performance Metrics as Predictors of Early Urinary Continence Recovery After Robotic Radical Prostatectomy—a Prospective Bi-institutional Study,” European Urology Open Science, May… [cited by applicant]
“Imbalanced-learn API”, API webpage [URL: https://imbalanced-learn.readthedocs.io/en/stable/api.html], as of Nov. 24, 2020. Retrieved from Internet Archive Wayback Machine [URL: https://web.archive.org/web/2020112410005… [cited by applicant]
International Preliminary Report on Patentability for Application No. PCT/US2021/060900 mailed Jun. 15, 2023, 11 pages. [cited by applicant]
International Search Report and Written Opinion for Application No. PCT/US2021/060900, mailed Apr. 20, 2022, 16 pages. [cited by applicant]
International Search Report and Written Opinion for Application No. PCT/US2022/026080, mailed on Aug. 3, 2022, 13 pages. [cited by applicant]
Invitation to Pay Additional Fees and Partial International Search Report for PCT/US2021/060900, mailed Feb. 25, 2022, 11 pages. [cited by applicant]
James, G., et al., “An Introduction to Statistical Learning with Applications in R,” online PDF, indicated as corresponding to “An Introduction to Statistical Learning with Applications in R,” Springer Science+Business … [cited by applicant]
Jarc. A., et al., “Viewpoint matters: objective performance metrics for surgeon endoscope control during robot-assisted surgery” as received in connection with the International Search Report and Written Opinion for App… [cited by applicant]
Kloek, C.E., et al., “A Broadly Applicable Surgical Teaching Method: Evaluation of a Stepwise Introduction to Cataract Surgery,” Journal of Surgical Education, 2014, vol. 71 (2), pp. 169-175. Retrieved from an account w… [cited by applicant]
Laurikkala, J., “Improving Identification of Difficult Small Classes by Balancing Class Distribution,” Proceedings of the 8th Conference on AI in Medicine in Europe: Artificial Intelligence Medicine, Jul. 2001, pp. 63-6… [cited by applicant]
Lazar, J., et al., “Objective Performance Indicators of Cardiothoracic Residents Are Associated with Vascular Injury During Robotic-Assisted Lobectomy on Porcine Models” as received in connection with the International … [cited by applicant]
Lecuyer, G., et al., “Assisted phase and step annotation for surgical videos,” understood to correspond to International Journal of Computer Assisted Radiology and Surgery, Apr. 2020, vol. 15 (4), pp. 673-680 though ind… [cited by applicant]
Lemaitre, G., et al., “Imbalanced-learn: A Python Toolbox to Tackle the Curse of Imbalanced Datasets in Machine Learning,” Journal of Machine Learning Research, 2017, vol. 18, pp. 1-5. Retrieved from [URL: https://www.j… [cited by applicant]
Liu, M., et al., “Assessment of Robotic Console Skills (ARCS): construct validity of a novel global rating scale for technical skills in robotically assisted surgery,” understood to correspond to Surgical Endoscopy, Jan… [cited by applicant]
Loukas, C., “Video content analysis of surgical procedures,” Surgical Endoscopy, Feb. 2018, vol. 32(2), pp. 553-568. Retrieved from an account with “Reprints Desk” [URL: https://www.researchsolutions.com/] in Oct. 2023,… [cited by applicant]
Lyman, W.B., et al., “Novel Objective Approach to Evaluate Novice Robotic Surgeons Using a Combination of Kinematics and Stepwise Cumulative Sum Analyses,” Journal of the American College of Surgeons, Oct. 2018, vol. 22… [cited by applicant]
Maier-Hein, L., et al., “Surgical Data Science—from Concepts toward Clinical Translation,” Medical Image Analysis, Feb. 2022, 76:102306, pp. 1-95. Retrieved from [URL: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC913505… [cited by applicant]
Maier-Hein, L., et al., “Surgical data science: enabling next-generation surgery.” arXiv preprint arXiv:1701.06482 (2017). Per [URL: https://spiral.imperial.ac.uk/handle/10044/1/62866] retrieved Oct. 25, 2023, understoo… [cited by applicant]
Matthews, B.W., “Comparison of the Predicted and Observed Secondary Structure of T4 Phage Lysozyme,” Biochimica et Biophysica Acta, Oct. 1975, vol. 405(2), pp. 442-451. Retrieved from an account with “Reprints Desk” [UR… [cited by applicant]
Ozdemir-Van Brunschot, D.M.D., et al., “Surgical Team Composition Has a Major Impact on Effectiveness and Costs in Laparoscopic Donor Nephrectomy,” World Journal of Urology, May 2015, vol. 33(5), pp. 733-741. Retrieved … [cited by applicant]
Padoy, N., “Machine and deep learning for workflow recognition during surgery,” Minimally Invasive Therapy & Allied Technologies, Apr. 2019, vol. 28 (2), pp. 82-90. Retrieved from an account with “Reprints Desk” [URL: h… [cited by applicant]
Peabody, J.O., et al., “Wisdom of the Crowds: Use of Crowdsourcing to Assess Surgical Skill of Robot-Assisted Radical Prostatectomy in a Statewide Surgical Collaborative” European Urology Supplements, Apr. 2015, vol. 14… [cited by applicant]
Pearce, S.M., et al., “The Impact of Days off Between Cases on Perioperative Outcomes for Robotic-assisted Laparoscopic Prostatectomy,” understood to correspond to World Journal of Urology, Feb. 2016, vol. 34(2), pp. 26… [cited by applicant]
Pedregosa, F., et al., “Scikit-learn: Machine Learning in Python,” Journal of Machine Learning Research, 2011, vol. 12, pp. 2825-2830. Retrieved from [URL: https://www.jmlr.org/papers/volume12/pedregosa11a/pedregosa11a.… [cited by applicant]
Powers, M.K., et al., “Crowdsourcing Assessment of Surgeon Dissection of Renal Artery and Vein During Robotic Partial Nephrectomy: A Novel Approach for Quantitative Assessment of Surgical Performance,” Journal of Endour… [cited by applicant]
Raad, W.N., et al., “Robotic Thoracic Surgery Training for Residency Programs: a Position Paper for an Educational Curriculum,” Innovations, Dec. 2018, vol. 13 (6), pp. 417-422. Retrieved from an account with “Reprints … [cited by applicant]
Reiley, C.E., et al., “Review of Methods for Objective Surgical Skill Evaluation,” Surgical Endoscopy, Feb. 2011, vol. 25, pp. 356-366. Retrieved from an account with “Reprints Desk” [URL: https://www.researchsolutions.… [cited by applicant]
Sarikaya, D. and Jannin, P., “Surgical Activity Recognition Using Learned Spatial Temporal Graph Representations of Surgical Tools,” arXiv preprint arXiv:2001.03728 (2020). Indicated as corresponding to IEEE Transaction… [cited by applicant]
Satava, R.M., et al., “Proving the Effectiveness of the Fundamentals of Robotic Surgery (FRS) Skills Curriculum: A Single-blinded, Multispecialty, Multi-institutional Randomized Control Trial,” Understood to correspond … [cited by applicant]
Scott, D. “Multivariate Density Estimation and Visualization” online PDF, indicated as posted upon Jun. 16, 2007 at [URL: http://yaroslavvb.com/papers/] as acquired Aug. 16, 2023. Acquired from [URL: http://yaroslavvb.c… [cited by applicant]
Silverman, B.W., “Density Estimation for Statistics and Data Analysis,” understood to correspond to Monographs on Statistics and Applied Probability, 1986, pp. 1-22. Retrieved from [URL: https://ned.ipac.caltech.edu/lev… [cited by applicant]
Van Amsterdam, B., et al., “Weakly Supervised Recognition of Surgical Gestures,” IEEE International Conference on Robotics and Automation, Jul. 2019, pp. 9565-9571. Retrieved from an account with “Reprints Desk” [URL: h… [cited by applicant]
Vedula, S.S., et al., “Objective Assessment of Surgical Technical Skill and Competency in the Operating Room,” Annual Review of Biomedical Engineering, Jun. 2017, vol. 19, pp. 301-325. Retrieved from [URL: https://www.n… [cited by applicant]
Vertut, J., and Coiffet, P., “Robot Technology: Teleoperation and Robotics Evolution and Development,” English translation, Prentice-Hall, Inc., Inglewood Cliffs, NJ, USA 1986, vol. 3A. 332 pages. [cited by applicant]
Virtanen, P., et al., “SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python,” Nature Methods, Mar. 2020, vol. 17, pp. 261-272. Retrieved from [URL: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7056644/pd… [cited by applicant]
Zia, A. and Essa, I. “Automated Surgical Skill Assessment in RMIS Training” arXiv preprint arXiv: 1712.08604 (2017). Retrieved from [URL: https://arxiv.org/pdf/1712.08604.pdf] on Aug. 18, 2023, 12 pages. [cited by applicant]
Zia, A., et al., “Novel Evaluation of Surgical Activity Recognition Models Using Task-Based Efficiency Metrics,” arXiv preprint arXiv: 1712.08604 (2019). Retrieved from [URL: https://arxiv.org/pdf/1907.02060.pdf] on Oct… [cited by applicant]
Zia, A., et al., “Temporal clustering of surgical activities in robot-assisted surgery,” International Journal of Computer Assisted Radiology and Surgery, Jul. 2017, vol. 12 (7), pp. 1171-1178. Retrieved from [URL: http… [cited by applicant]
Zia, A., et al. “Surgical Activity Recognition in Robot-Assisted Radical Prostatectomy Using Deep Learning.” Medical Image Computing and Computer Assisted Intervention—MICCAI 2018: 21st International Conference, Granada… [cited by applicant]
International Search Report and Written Opinion for Application No. PCT/US2024/028840, mailed on Sep. 16, 2024, 14 pages. [cited by applicant]