IP Library › Granted Patent US 12,592,312
Granted Patent B2
US 12,592,312 · App. 17/384,337 · Granted Mar 31, 2026

Surgical data system and control

Inventors: Frederick E. Shelton, IV (Hillsboro, OH); Kevin Fiebig (Cincinnati, OH)
Assignee: Cilag GmbH International
G16H40/20A61B17/00A61B18/1206A61B34/10A61B34/20A61B34/25A61B34/30A61B34/32A61B90/08A61B90/37G05B13/0265G06F3/14G06F3/1423G06F3/167G06F9/4881G06F9/542G06F13/4068G06F16/211G06F16/284G06F16/285G06N20/00G06Q10/30G06T11/60G08B5/22G10L15/22G16H10/60G16H15/00G16H20/40G16H30/40G16H40/40G16H40/63G16H40/67G16H50/20G16H50/70H04L1/22H04L41/12H04L65/80H04L67/12H04L67/125H04N5/272H04N7/15A61B8/06A61B2017/00221A61B2018/00702A61B2018/00994A61B2034/2072A61B2034/254A61B2090/364A61B2090/365A61B2090/373G06F21/6245G06F40/169G10L2015/223G16H30/20H02J7/0063
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,592,312
App. No.
17/384,337
Granted
Mar 31, 2026
Kind
B2
Abstract

A device to process data associated with a surgical event of a surgery may include a processor. The processor may be configured to receive multiple data streams during the surgical event. The processor may be configured to select a primary data stream based on a surgical data interface via which the primary data stream is received. The processor may be configured to select a secondary data stream based on a surgical data interface via which the second data stream is received. The processor may be configured to identify the surgical data interfaces. The processor may be configured to generate situational data associated with the primary data stream based on the secondary data stream. The situational data may indicate a medical decision-making factor of the surgical event. The primary data stream and the situational data may be sent during the surgical event.

Claims (43)

1 . A device comprising:

a processor configured to:

receive a plurality of data streams during a surgical event, wherein the plurality of data streams comprises a first data stream and a second data stream;

identify a first surgical data interface via which the first data stream is received, wherein the first surgical data interface is configured to receive the first data stream from a first surgical instrument;

identify a second surgical data interface via which the second data stream is received, wherein the second surgical data interface is configured to receive the second data stream from a second surgical instrument;

select the first data stream and the second data stream from the plurality of data streams;

determine the first data stream as a primary data stream based on the first surgical data interface configured to receive the first data stream from the first surgical instrument;

determine the second data stream as a secondary data stream based on the second surgical data interface configured to receive the second data stream from the second surgical instrument;

generate, based on the secondary data stream, situational data associated with the primary data stream, wherein the situational data indicates a medical decision-making factor of the surgical event;

send the primary data stream with the situational data associated with the primary data stream during the surgical event;

generate control instructions based on the primary data stream and the situational data associated with the primary data stream; and

send the control instructions to a third surgical instrument in communication with the device, wherein the generated control instructions change an operation of the third surgical instrument.

2 . The device of claim 1 , wherein the first data stream comprises a first timing element, the first timing element indicates a first time when the first data stream is collected during the surgical event, the primary data stream is sent with the situational data associated with the primary data stream at a second time during the surgical event, and a difference between the first time and the second time is lower than a predetermined value.

3 . The device of claim 2 , wherein the predetermined value is associated with real-time processing, and the difference being lower than the predetermined value indicates that the collection of the first data stream and the sending of the primary data stream occur in real time.

4 . The device of claim 1 , wherein the primary data stream is sent via data packets, and wherein at least one of the data packets comprises a field indicative of the situational data.

5 . The device of claim 1 , wherein the first surgical data interface is designated to communicate with a first type of surgical instrument, and wherein the second surgical data interface is designated to communicate with a second type of surgical instrument.

6 . The device of claim 1 , wherein the situational data is sent using at least one of an annotation for the primary data stream, a context associated with the primary data stream, or meta data that indicates the context associated with the primary data stream.

7 . The device of claim 1 , wherein the processor is further configured to:

generate a risk indicator based on the primary data stream and the situational data associated with the primary data stream; and

send the risk indicator.

8 . The device of claim 7 , wherein the risk indicator comprises at least one of an action trigger, a notification, or a threshold.

9 . The device of claim 1 , wherein the secondary data stream of the plurality of data streams comprises a first portion and a second portion, and the processor is further configured to store one of the first portion or the second portion of the secondary data stream.

10 . The device of claim 1 , wherein the surgical event comprises an ongoing surgical event, the plurality of data streams comprises a third data stream associated with a historical surgical event, and the processor is further configured to determine that the ongoing surgical event and the historical surgical event have a characteristic in common, and wherein the situational data is generated further based on the third data stream associated with the historical surgical event.

11 . The device of claim 10 , wherein the characteristic in common comprises at least one of a same patient, a same type of surgical procedure, a same type of surgical instrument, or a same type of surgical equipment.

12 . A method performed by a device with a processor, the method comprising:

receiving, via the processor, a plurality of data streams during a surgical event, wherein the plurality of data streams comprises a first data stream and a second data stream;

identifying, via the processor, a first surgical data interface via which the first data stream is received, wherein the first surgical data interface is configured to receive the first data stream from a first surgical instrument;

identifying, via the processor, a second surgical data interface via which the second data stream is received, wherein the second surgical data interface is configured to receive the second data stream from a second surgical instrument;

selecting, via the processor, the first data stream and the second data stream from the plurality of data streams;

determining, via the processor, the first data stream as a primary data stream based on the first surgical data interface configured to receive the first data stream from the first surgical instrument;

determining, via the processor, the second data stream as a secondary data stream based on the second surgical data interface configured to receive the second data stream from the second surgical instrument;

generating, via the processor, based on the secondary data stream, situational data associated with the primary data stream, wherein the situational data indicates a medical decision-making factor of the surgical event;

sending, via the processor, the primary data stream with the situational data associated with the primary data stream during the surgical event;

generating, via the processor, control instructions based on the primary data stream and the situational data associated with the primary data stream; and

sending, via the processor, the control instructions to a third surgical instrument in communication with the device, wherein the generated control instructions change an operation of the third surgical instrument.

13 . The method of claim 12 , wherein the first data stream comprises a first timing element, the first timing element indicates a first time when the first data stream is collected during the surgical event, the primary data stream is sent with the situational data associated with the primary data stream at a second time during the surgical event, and a difference between the first time and the second time is lower than a predetermined value.

14 . The method of claim 13 , wherein the predetermined value is associated with real-time processing, and the difference being lower than the predetermined value indicates that the collection of the first data stream and the sending of the primary data stream occur in real time.

15 . The method of claim 12 , further comprising:

generating a risk indicator based on the primary data stream and the situational data associated with the primary data stream; and

sending the risk indicator.

16 . The method of claim 12 , wherein the secondary data stream of the plurality of data streams comprises a first portion and a second portion, and the method further comprising storing one of the first portion or the second portion of the secondary data stream.

17 . The method of claim 12 , wherein the surgical event comprises an ongoing surgical event, the plurality of data streams comprises a third data stream associated with a historical surgical event, and the method further comprises determining that the ongoing surgical event and the historical surgical event have a characteristic in common, and wherein the situational data is generated further based on the third data stream associated with the historical surgical event.

18 . The method of claim 17 , wherein the characteristic in common comprises at least one of a same patient, a same surgical procedure, a same surgical instrument, or a same surgical equipment.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 8, 2021
From: SHELTON, FREDERICK E., IV; FIEBIG, KEVIN
To: CILAG GMBH INTERNATIONAL
Reel/Frame 057738/0848 →
Continuity (2)
Provisional Application 63224813 · Jul 22, 2021
Related Publication 20230027210A1 · Jan 26, 2023
References Cited (131)
US 6766373B1 · Beadle et al. · 2004 [cited by applicant]
US 8565073B2 · Rahman et al. · 2013 [cited by applicant]
US 8908678B1 · Mcgonigal et al. · 2014 [cited by applicant]
US 9011427B2 · Price et al. · 2015 [cited by applicant]
US 9283054B2 · Morgan et al. · 2016 [cited by applicant]
US 9345481B2 · Hall et al. · 2016 [cited by applicant]
US 11146690B2 · Minert · 2021 [cited by applicant]
US 11232868B1 · Sutherland et al. · 2022 [cited by applicant]
US 11564573B2 · Hirst · 2023 [cited by applicant]
US 20020000464A1 · Ramberg et al. · 2002 [cited by applicant]
US 20050210070A1 · Macneil · 2005 [cited by applicant]
US 20110276340A1 · Deboer et al. · 2011 [cited by applicant]
US 20130051220A1 · Ryshakov · 2013 [cited by applicant]
US 20130149967A1 · Ma et al. · 2013 [cited by applicant]
US 20140160259A1 · Blanquart et al. · 2014 [cited by applicant]
US 20140160260A1 · Blanquart et al. · 2014 [cited by applicant]
US 20140163927A1 · Molettiere et al. · 2014 [cited by applicant]
US 20140263552A1 · Hall et al. · 2014 [cited by applicant]
US 20140267655A1 · Richardson et al. · 2014 [cited by applicant]
US 20150119035A1 · Ganu et al. · 2015 [cited by applicant]
US 20150128274A1 · Giokas · 2015 [cited by applicant]
US 20150182118A1 · Bradbury et al. · 2015 [cited by applicant]
US 20150215159A1 · Liao et al. · 2015 [cited by applicant]
US 20170006135A1 · Siebel et al. · 2017 [cited by applicant]
US 20170085627A1 · Goldstein et al. · 2017 [cited by applicant]
US 20170296213A1 · Swensgard et al. · 2017 [cited by applicant]
US 20180122506A1 · Grantcharov · 2018 [cited by examiner]
US 20180344308A1 · Nawana et al. · 2018 [cited by applicant]
US 20180360452A1 · Shelton, IV et al. · 2018 [cited by applicant]
US 20190125455A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190191963A1 · Kuhn et al. · 2019 [cited by applicant]
US 20190200844A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190200906A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190200980A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190200988A1 · Shelton, IV · 2019 [cited by applicant]
US 20190201033A1 · Yates et al. · 2019 [cited by applicant]
US 20190201102A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190201104A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190201115A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190201123A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190201124A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190201125A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190201126A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190201127A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190201129A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190201137A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190201140A1 · Yates et al. · 2019 [cited by applicant]
US 20190204201A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190205441A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190205566A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190205567A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190206216A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190206542A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190206551A1 · Yates et al. · 2019 [cited by applicant]
US 20190206556A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190206562A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190206569A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190206576A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190207773A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190207857A1 · Shelton, IV et al. · 2019 [cited by applicant]
US 20190207911A1 · Wiener et al. · 2019 [cited by applicant]
US 20190333626A1 · Mansi · 2019 [cited by examiner]
US 20200244734A1 · Mendiola et al. · 2020 [cited by applicant]
US 20200405403A1 · Shelton, IV et al. · 2020 [cited by applicant]
US 20210145523A1 · Xing et al. · 2021 [cited by applicant]
US 20210205030A1 · Shelton, IV et al. · 2021 [cited by applicant]
US 20210205031A1 · Shelton, IV · 2021 [cited by applicant]
US 20210212717A1 · Yates · 2021 [cited by examiner]
US 20210290046A1 · Nazareth et al. · 2021 [cited by applicant]
US 20210313051A1 · Asselmann et al. · 2021 [cited by applicant]
US 20210319894A1 · Sobol et al. · 2021 [cited by applicant]
US 20210346094A1 · Fuerst et al. · 2021 [cited by applicant]
US 20220020476A1 · Souissi · 2022 [cited by applicant]
US 20220020486A1 · Giataganas et al. · 2022 [cited by applicant]
US 20220046292A1 · Nair et al. · 2022 [cited by applicant]
US 20220104713A1 · Shelton, IV · 2022 [cited by applicant]
US 20220104807A1 · Shelton, IV et al. · 2022 [cited by applicant]
US 20220104896A1 · Shelton, IV et al. · 2022 [cited by applicant]
US 20220104910A1 · Shelton, IV et al. · 2022 [cited by applicant]
US 20220108789A1 · Shelton, IV et al. · 2022 [cited by applicant]
US 20220233119A1 · Shelton, IV et al. · 2022 [cited by applicant]
US 20220233135A1 · Shelton, IV et al. · 2022 [cited by applicant]
US 20220233136A1 · Shelton, IV et al. · 2022 [cited by applicant]
US 20220233151A1 · Shelton, IV et al. · 2022 [cited by applicant]
US 20220233191A1 · Shelton, IV et al. · 2022 [cited by applicant]
US 20220233252A1 · Shelton, IV et al. · 2022 [cited by applicant]
US 20220233254A1 · Shelton, IV et al. · 2022 [cited by applicant]
US 20220238216A1 · Shelton, IV et al. · 2022 [cited by applicant]
US 20220240869A1 · Shelton, IV et al. · 2022 [cited by applicant]
US 20220241028A1 · Shelton, IV et al. · 2022 [cited by applicant]
US 20220241474A1 · Shelton, IV et al. · 2022 [cited by applicant]
US 20220303945A1 · Tsuda · 2022 [cited by applicant]
US 20230021832A1 · Shelton, IV et al. · 2023 [cited by applicant]
US 20230021920A1 · Shelton, IV et al. · 2023 [cited by applicant]
US 20230022604A1 · Shelton, IV · 2023 [cited by applicant]
US 20230023635A1 · Shelton, IV et al. · 2023 [cited by applicant]
US 20230025790A1 · Shelton, IV et al. · 2023 [cited by applicant]
US 20230025827A1 · Shelton, IV et al. · 2023 [cited by applicant]
US 20230026893A1 · Shelton, IV et al. · 2023 [cited by applicant]
US 20230027543A1 · Shelton, IV et al. · 2023 [cited by applicant]
US 20230028059A1 · Shelton, IV et al. · 2023 [cited by applicant]
US 20230028633A1 · Shelton, IV et al. · 2023 [cited by applicant]
US 20230028677A1 · Shelton, IV et al. · 2023 [cited by applicant]
US 20230035775A1 · Kohada · 2023 [cited by applicant]
US 20230293236A1 · Wright et al. · 2023 [cited by applicant]
EP 3506287A1 · 2019 [cited by applicant]
JP 2017504019A5 · 2017 [cited by applicant]
WO 2013174327A1 · 2013 [cited by applicant]
WO 2017089479A1 · 2017 [cited by applicant]
WO 2019119130A1 · 2019 [cited by applicant]
WO 2020159978A1 · 2020 [cited by applicant]
WO 2021048326A1 · 2021 [cited by applicant]
WO 2022195306A1 · 2022 [cited by applicant]
Hashimoto, et al., “Artificial Intelligence in Surgery: Promises and Perils”, Annals of Surgery, vol. 268, No. 1, Jul. 2018, pp. 70-76. [cited by applicant]
Jagannath, et al., “An Analysis of Speech as a Modality for Activity Recognition during Complex Medical Teamwork”, Pervasive Computing Technologies for Healthcare, May 2018, pp. 1-10. [cited by applicant]
Ban, Yutong et al., “Aggregating Long-Term Context for Learning Laparoscopic and Robot-Assisted Surgical Workflows”, 2021 IEEE International Conference on Robotics and Automation (ICRA 2021), May 31-Jun. 4, 2021, 8 page… [cited by applicant]
Kum, Sang-Uok , “Encoding of Multiple Depth Streams”, 2008, 162 pages. [cited by applicant]
Amsterdam, et al., “Gesture Recognition in Robotic Surgery: A Review”, IEEE Transactions on Biomedical Engineering, vol. 68, No. 6, Jun. 2021, pp. 2021-2035. [cited by applicant]
Cugola, et al., “Processing Flows of Information: From Data Stream to Complex Event Processing”, ACM Computing Surveys, vol. 44, No. 03, Jun. 2012, 71 pages. [cited by applicant]
ISO 26262-1, “Road Vehicles—Functional Safety—Part 1: Vocabulary”, International Organization for Standardization, Edition 2, Dec. 2018, 13 pages. [cited by applicant]
Khan, et al., “Zeqos: A New Energy and QoS-Aware Routing Protocol for Communication of Sensor Devices in Healthcare System”, International Journal of Distributed Sensor Networks, vol. 10, No. 06, Jun. 5, 2014, 24 pages. [cited by applicant]
Namazi, et al., “Automatic Detection of Surgical Phases in Laparoscopic Videos”, Proceedings on the international conference in artificial intelligence (ICAI), Aug. 2018, pp. 124-130. [cited by applicant]
International Search Report and Written Opinion, Received for PCT Application No. PCT/IB2022/056665, mailed on Oct. 31, 2022, 17 pages. [cited by applicant]
International Search Report and Written Opinion, Received for PCT Application No. PCT/IB2022/056666, mailed on Oct. 31, 2022, 11 pages. [cited by applicant]
International Search Report and Written Opinion, received for PCT Application No. PCT/IB2022/056669, mailed on Oct. 10, 2022, 16 pages. [cited by applicant]
Rahmani, et al., “Exploiting Smart E-Health Gateways at the Edge of Healthcare Internet-of-Things: A Fog Computing Approach”, Future Generation Computer Systems, vol. 78, No. 02, Feb. 2017, 47 pages. [cited by applicant]
Ren, et al., “Multisensor Data Fusion in an Integrated Tracking System for Endoscopic Surgery”, IEEE Transactions on Information Technology In Biomedicine, vol. 16. No. 1, Jan. 2012, pp. 106-111. [cited by applicant]
Doan, et al., “Introduction”, Principles of Data Integration, Chapter 01, 2012, pp. 1-18. [cited by applicant]
Doan, et al., “Schema Matching and Mapping”, Principles of Data Integration, Chapter 05, 2012, pp. 121-160. [cited by applicant]
Guibert, et al., “CC-Fog: Toward Content-Centric Fog Networks for E-Health”, IEEE 19th International Conference on e-Health Networking, Applications and Services (Healthcom), Oct. 12-15, 2017, 5 pages. [cited by applicant]
“Blockchain”, Wikipedia, (https://en.wikipedia.org/wiki/Blockchain), Jul. 2021, pp. 1-34. [cited by applicant]