Segmentation generation for image-related functions
A computer system may perform an image-related function using a segmentation (e.g., an image mask) that has been generated by a custom segmentation machine learning model. To begin, the system may receive image data corresponding to a surgical scene including a background that includes an anatomical feature and at least one surgical tool. The system may also generate segmentation data using the custom segmentation machine learning model based on inputting the first image data to the custom segmentation machine learning model. The system may also include generating a segmentation of the at least one surgical tool using the segmentation data. Once the segmentation has been generated, the system may perform an image-related function using the segmentation.
1 . A computer-implemented method, comprising:
receiving first image data corresponding to a surgical scene comprising a background that includes an anatomical feature and at least one surgical tool;
generating segmentation data using a custom segmentation model based on inputting the first image data to the custom segmentation model, wherein the custom segmentation model is based on:
a first set of segmentations output from a first machine learning model based on a first set of inputs comprising (i) first images of a set of real images of a first set of surgical scenes, and (ii) a set of bounding boxes identifying a set of real surgical tools in the first images of the set of real images;
a set of domain adapted synthetic images output from a second machine learning model based on a set of synthetic images of a second set of surgical scenes output from a simulation tool, wherein the simulation tool takes as input a second set of inputs comprising (i) three-dimensional models of surgical tools, (ii) three-dimensional models of organs, and (iii) second images of the first set of real images that exclude surgical tools; and
a second set of segmentations output from the simulation tool and corresponding to the set of synthetic images;
generating a segmentation of the at least one surgical tool using the segmentation data; and
performing an image-related function using the segmentation.
2 . The computer-implemented method of claim 1 , wherein the first machine learning model comprises a neural network architecture configured for partially supervised segmentation tasks.
3 . The computer-implemented method of claim 2 , wherein the neural network architecture comprises a DeepMAC model.
4 . The computer-implemented method of claim 1 , wherein the segmentation comprises a binary image mask that identifies the at least one surgical tool and excludes the background.
5 . The computer-implemented method of claim 1 , wherein a first subset of synthetic images of the set of synthetic images are based at least in part on a subset of the three-dimensional models of organs representing synthetic backgrounds and a subset of the three-dimensional models of the surgical tool.
6 . The computer-implemented method of claim 5 , wherein a second subset of synthetic images of the set of synthetic images is based at least in part on the second images of the first set of real images representing real backgrounds and a second subset of the three-dimensional models of the surgical tool.
7 . The computer-implemented method of claim 1 , further comprising training the first machine learning model by at least:
generating the first set of segmentations by inputting the first images of the set of real images and the set of bounding boxes into the first machine learning model;
providing the first set of segmentations and the first images of the set of real images as first training data for the custom segmentation model;
generating the set of synthetic images by inputting the second images of the first set of real images and a subset of the three-dimensional models of surgical tools into the simulation tool;
generating the second set of segmentations by inputting the subset of the three-dimensional models of surgical tools into the simulation tool; and
providing the set of synthetic images and the second set of segmentations as second training data for the custom segmentation model.
8 . The computer-implemented method of claim 7 , wherein generating the set of synthetic images further comprises generating adapted versions of the set of synthetic images by inputting the set of synthetic images into the second machine learning model that adapts the set of synthetic images to a surgical domain to define the set of domain adapted synthetic images, wherein providing the set of synthetic images as the second training data comprises providing the set of domain adapted synthetic images.
9 . The computer-implemented method of claim 8 , wherein the custom segmentation model comprises a custom-trained segmentation model, the first machine learning model comprises a DeepMAC model, the second machine learning model comprises a CycleGAN model, and the simulation tool comprises Blender3D.
10 . A non-transitory computer-readable storage device comprising computer-executable instructions that, when executed by a computer system, cause the computer system to perform operations comprising:
receiving first image data corresponding to a surgical scene comprising a background that includes an anatomical feature and at least one surgical tool;
generating segmentation data using a custom segmentation model based on inputting the first image data to the custom segmentation model, wherein the custom segmentation model is based on:
a first set of segmentations output from a first machine learning model based on a first set of inputs comprising (i) first images of a set of real images of a first set of surgical scenes, and (ii) a set of bounding boxes identifying a set of real surgical tools in the first images of the set of real images;
a set of domain adapted synthetic images output from a second machine learning model based on a set of synthetic images of a second set of surgical scenes output from a simulation tool, wherein the simulation tool takes as input a second set of inputs comprising (i) three-dimensional models of surgical tools, (ii) three-dimensional models of organs, and (iii) second images of the first set of real images that exclude surgical tools; and
a second set of segmentations output from the simulation tool and corresponding to the set of synthetic images;
generating a segmentation of the at least one surgical tool using the segmentation data; and
performing an image-related function using the segmentation.
11 . The non-transitory computer-readable storage device of claim 10 , wherein the first set of surgical scenes are real surgical scenes, and the second set of surgical scenes are synthetic surgical scenes.
12 . The non-transitory computer-readable storage device of claim 10 , wherein each segmentation of the second set of segmentations represents at least one simulated surgical tool of at least one synthetic image of the set of synthetic images.
13 . The non-transitory computer-readable storage device of claim 10 , wherein the set of synthetic images and the second set of segmentations are generated by the simulation tool.
14 . The non-transitory computer-readable storage device of claim 10 , wherein the set of synthetic images are adapted to a surgical domain using the second machine learning model that comprises a deep convolutional neural network.
15 . The non-transitory computer-readable storage device of claim 14 , wherein the deep convolutional neural network comprises a Cycle Generative Adversarial Network.
16 . The non-transitory computer-readable storage device of claim 10 , wherein each segmentation of the first set of segmentations represents at least one real surgical tool of at least one real image of the first images of the set of real images.
17 . The non-transitory computer-readable storage device of claim 10 , wherein the first image data corresponds to an image of the surgical scene, and wherein performing the image-related function comprises adjusting a characteristic of the image using the segmentation.
18 . The non-transitory computer-readable storage device of claim 17 , wherein the image is a frame in a video feed from a laparoscopic camera that is viewing the surgical scene.
19 . The non-transitory computer-readable storage device of claim 17 , wherein the at least one surgical tool is in use during a surgical procedure upon the anatomical feature.
20 . A system, comprising:
a memory configured to store computer-executable instructions; and
a processor configured to access the memory and execute the computer-executable instructions to at least:
receive first image data corresponding to a surgical scene comprising a background that includes an anatomical feature and at least one surgical tool;
generate segmentation data using a custom segmentation model based on inputting the first image data to the custom segmentation model, wherein the custom segmentation model is based on:
a first set of segmentations output from a first machine learning model based on a first set of inputs comprising (i) first images of a set of real images of a first set of surgical scenes, and (ii) a set of bounding boxes identifying a set of real surgical tools in the first images of the set of real images;
a set of domain adapted synthetic images output from a second machine learning model based on a set of synthetic images of a second set of surgical scenes output from a simulation tool, wherein the simulation tool takes as input a second set of inputs comprising (i) three-dimensional models of surgical tools, (ii) three-dimensional models of organs, and (iii) second images of the first set of real images that exclude surgical tools; and
a second set of segmentations output from the simulation tool and corresponding to the set of synthetic images;
generate a segmentation of the at least one surgical tool using the segmentation data; and
perform an image-related function using the segmentation.
21 . The system of claim 20 , wherein performing the image-related function comprises:
generating a record of use of the at least one surgical tool using the segmentation; and
recording the record of use in a database.
22 . The system of claim 20 , wherein performing the image-related function comprises using the segmentation to estimate a cost of a surgical procedure that uses the at least one surgical tool.
23 . The system of claim 20 , wherein the system is a remote computing system, wherein receiving the first image data comprises receiving, by the remote computing system, the first image data from a local computing device.
24 . The system of claim 23 , wherein the memory is configured to store additional computer-executable instructions that, when executed by the processor, cause the system to at least send, by the remote computer system, second image data to the local computing device, and wherein the second image data corresponds to an image of the surgical scene that has been modified by the segmentation.
25 . The system of claim 20 , wherein the segmentation comprises a pixel-accurate segmentation.