Systems and methods for displaying intraoperative image data
An exemplary method of displaying an intraoperative image of a surgical site comprises: receiving a plurality of images captured by a plurality of in-light cameras integrated into one or more surgical light units illuminating the surgical site, wherein the plurality of images capture the surgical site from a plurality of different perspectives; identifying an obstruction to the surgical site in an image of the plurality of images; responsive to identifying the obstruction, generating a composite image based on a set of the plurality of images, wherein the composite image excludes the obstruction; and displaying the composite image as the intraoperative image of the surgical site.
1 . A method of displaying an intraoperative image of a surgical site, comprising:
receiving a plurality of images captured by a plurality of in-light cameras integrated into a surgical light unit illuminating the surgical site, the surgical light unit comprising a plurality of sensors, wherein the plurality of images capture the surgical site from a plurality of different perspectives, wherein the plurality of in-light cameras of the surgical light unit comprises a central in-light camera protruding from the surgical light unit and one or more peripheral in-light cameras surrounding the central in-light camera;
automatically orienting an image captured by the central in-light camera such that a top of the image captured by the central in-light camera is aligned with a top of the surgical light unit based on information from one or more sensors of the plurality of sensors;
determining a presence of an obstruction to the surgical site in the image captured by the central in-light camera;
in accordance with a determination that the obstruction is not present, displaying the image captured by the central in-light camera as the intraoperative image of the surgical site; and
in accordance with a determination that the obstruction is present:
generating a composite image based on a set of the plurality of images, wherein the composite image excludes the obstruction; and
displaying the composite image as the intraoperative image of the surgical site.
2 . The method of claim 1 , wherein the plurality of images is captured at a same time.
3 . The method of claim 1 , comprising:
receiving a second plurality of images captured by a second plurality of in-light cameras integrated into a second surgical light unit illuminating the surgical site.
4 . The method of claim 3 , further comprising:
simultaneously displaying a first view corresponding to the first surgical light unit and a second view corresponding to the second surgical light unit, wherein the composite image is included in one of the first view and the second view.
5 . The method of claim 1 , wherein the obstruction includes a surgeon's head or a surgeon's body and does not include surgical tools or a surgeon's hand at the surgical site.
6 . The method of claim 1 , wherein determining the presence of the obstruction to the surgical site in the image comprises:
inputting the image into a trained machine-learning model to determine if the image contains an area representing the obstruction.
7 . The method of claim 6 , wherein the trained machine-learning model is configured to receive an input image and detect an area in the input image as blocking the surgical site.
8 . The method of claim 6 , wherein the machine-learning model is trained using a plurality of labelled training images.
9 . The method of claim 6 , wherein the machine learning model is trained by pre-training the machine learning model based on a plurality of training images that were not captured by in-light cameras and fine-tuning the machine learning model based on a plurality of training images captured by in-light cameras.
10 . The method of claim 1 , wherein determining the presence of the obstruction to the surgical site in the image comprises:
inputting the image into a trained machine-learning model to determine if the image contains the surgical site; and
if the image does not contain the surgical site, detecting the obstruction in the image.
11 . The method of claim 10 , wherein the trained machine-learning model is configured to receive an input image and detect an area in the input image as the surgical site.
12 . The method of claim 10 , wherein the machine-learning model is trained using a plurality of labelled training images.
13 . The method of claim 1 , wherein each in-light camera of the plurality of in-light cameras is associated with an in-light sensor in the same surgical light unit.
14 . The method of claim 13 , wherein determining the presence of the obstruction to the surgical site in the image comprises:
obtaining a proximity measurement of an in-light sensor associated with the in-light camera that captured the image;
comparing the proximity measurement with a predefined threshold; and
based on the comparison, determining whether the image includes the obstruction.
15 . The method of claim 14 , wherein the in-light sensor comprises a capacitive sensor, a Doppler sensor, an inductive sensor, a magnetic sensor, an optical sensor, a LiDAR sensor, a sonar sensor, an ultrasonic sensor, a radar sensor, or a hall effect sensor.
16 . The method of claim 1 , wherein determining the presence of the obstruction to the surgical site in the image comprises:
obtaining an auto-focus distance of the in-light camera that captured the image;
comparing the auto-focus distance with a predefined threshold; and
based on the comparison, determining whether the image includes the obstruction.
17 . The method of claim 1 , wherein determining the presence of the obstruction to the surgical site in the image comprises:
obtaining pixel-wise depth information of the image;
based on the pixel-wise depth information, determining whether the image includes the obstruction.
18 . The method of claim 1 , wherein generating the composite image based on the set of the plurality of images comprises: identifying, in an image from the plurality of images, an area representing the obstruction; and replacing pixels in the area with pixels from the set of images.
19 . The method of claim 1 , wherein at least one in-light camera of the plurality of in-light cameras comprises a fisheye lens.
20 . The method of claim 19 , further comprising: performing correction to an image captured by the at least one in-light camera to compensate for distortion caused by the fisheye lens.
21 . The method of claim 1 , further comprising: automatically reorienting one or more of the plurality of in-light cameras.
22 . The method of claim 1 , further comprising: reorienting one or more of the plurality of in-light cameras based on a user input.
23 . The method of claim 1 , further comprising:
inputting the composite image into a trained machine-learning model to detect an issue; and
outputting an alert based on the issue.
24 . The method of claim 23 , wherein the issue comprises one or more of peripheral tissue damage, incorrect procedures, undiagnosed issues, and retained surgical bodies.
25 . The method of claim 1 , wherein the composite image is displayed as part of a video image transmission stream.
26 . A system for displaying an intraoperative image of a surgical site, comprising:
a display;
one or more processors;
a memory; and
one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:
receiving a plurality of images captured by a plurality of in-light cameras integrated into a surgical light unit illuminating the surgical site, the surgical light unit comprising a plurality of sensors, wherein the plurality of images capture the surgical site from a plurality of different perspectives, wherein the plurality of in-light cameras of the surgical light unit comprises a central in-light camera protruding from the surgical light unit and one or more peripheral in-light cameras surrounding the central in-light camera;
automatically orienting an image captured by the central in-light camera such that a top of the image captured by the central in-light camera is aligned with a top of the surgical light unit based on information from one or more sensors of the plurality of sensors;
determining a presence of an obstruction to the surgical site in the image captured by the central in-light camera;
in accordance with a determination that the obstruction is not present, displaying the image captured by the central in-light camera as the intraoperative image of the surgical site; and
in accordance with a determination that the obstruction is present:
generating a composite image based on a set of the plurality of images, wherein the composite image excludes the obstruction; and
displaying the composite image as the intraoperative image of the surgical site.