Surgical simulator providing labeled data
A surgical simulator for simulating a surgical scenario comprises a display system, a user interface, and a controller. The controller includes one or more processors coupled to memory that stores instructions that when executed cause the system to perform operations. The operations include generating simulated surgical videos, each representative of the surgical scenario. The operations further include associating simulated ground truth data from the simulation with the simulated surgical videos. The ground truth data corresponds to context information of at least one of a simulated surgical instrument, a simulated anatomical region, a simulated surgical task, or a simulated action. The operations further include annotating features of the simulated surgical videos based, at least in part, on the simulated ground truth data for training a machine learning model.
1. A method comprising:
generating, with a surgical simulator, simulated surgical videos each representative of a simulation of a surgical scenario;
associating simulated ground truth data from the simulation with the simulated surgical videos, wherein the simulated ground truth data corresponds to context information of at least one of a simulated surgical instrument, a simulated anatomical region, or a simulated surgical task simulated by the surgical simulator;
annotating features of the simulated surgical videos based, at least in part, on the simulated ground truth data to generate annotated training data for training a machine learning model;
pre-training the machine learning model with the annotated training data to probabilistically identify the features:
providing, to a refiner neural network, simulated images from the simulated surgical videos before the pre-training;
refining the simulated surgical videos with the refiner neural network, wherein the refiner neural network adjusts the simulated images until a discriminator neural network determines the simulated images are comparable to unlabeled real images within a first threshold, wherein the features annotated from the simulated surgical videos are included after the refining, and wherein the unlabeled real images are representative of the surgical scenario in a real environment;
receiving unlabeled real videos from a surgical video database, each of the unlabeled real videos representative of the surgical scenario;
identifying the features from the unlabeled real videos with the machine learning model trained to probabilistically identify the features from the unlabeled real videos, wherein the features include at least one of a separation distance between a surgical instrument and an anatomical region, temporal boundaries of a surgical complication, or spatial boundaries of the surgical complication; and
annotating the unlabeled real videos with the machine learning model by labeling the features of the unlabeled real videos identified by the machine learning model.
2. The method of claim 1 , further comprising:
receiving, from the surgical video database, labeled real surgical videos, each representative of the surgical scenario in the real environment, and wherein the labeled real surgical videos include the features annotated for the machine learning model; and
after pre-training the machine learning model with the simulated surgical videos, training the machine learning model with the labeled real surgical videos to train the machine learning model to probabilistically identify the features from the unlabeled real videos.
3. The method of claim 1 , wherein the context information for each of the simulated surgical videos includes at least one of three-dimensional spatial boundaries of the simulated surgical instrument, three-dimensional spatial boundaries of the simulated anatomical region, motion of the simulated surgical instrument, separation distance between the simulated surgical instrument and the simulated anatomical region, orientation of the simulated surgical instrument, temporal boundaries of one or more surgical steps of the simulated surgical task from the surgical scenario, temporal boundaries of a simulated surgical complication, or spatial boundaries of the simulated surgical complication.
4. The method of claim 1 , further comprising:
rendering markers or identifiers superimposed on top of a corresponding one of the unlabeled real videos to generate annotated surgical videos.
5. The method of claim 1 , further comprising:
displaying annotated surgical videos corresponding to the unlabeled real videos with the features annotated by the machine learning model superimposed on a corresponding one of the unlabeled real videos.
6. The method of claim 1 , wherein the machine learning model includes at least one of a radial basis function neural network, a Kohonen self-organizing neural network, a recurrent neural network, a convolution neural network, or a modular neural network.
7. A non-transitory computer-readable storage medium having stored thereon instructions which, when executed by one or more processing units, cause the one or more processing units to perform operations comprising:
generating, with a surgical simulator, simulated surgical videos each representative of a simulation of a surgical scenario;
associating simulated ground truth data from the simulation with the simulated surgical videos, wherein the simulated ground truth data corresponds to context information of at least one of a simulated surgical instrument, a simulated anatomical region, or a simulated surgical task simulated by the surgical simulator;
annotating features of the simulated surgical videos based, at least in part, on the simulated ground truth data to generate annotated training data for training a machine learning model;
pre-training the machine learning model with the annotated training data to probabilistically identify the features;
receiving, from a surgical video database, labeled real surgical videos, each representative of the surgical scenario in a real environment, and wherein the labeled real surgical videos include the features annotated for the machine learning model; and
after the pre-training the machine learning model with the simulated surgical videos, training the machine learning model with the labeled real surgical videos to train the machine learning model to probabilistically identify the features from unlabeled real videos;
receiving the unlabeled real videos from the surgical video database, each of the unlabeled real videos representative of the surgical scenario;
identifying the features from the unlabeled real videos with the machine learning model trained to probabilistically identify the features from the unlabeled real videos, wherein the features include at least one of a separation distance between a surgical instrument and an anatomical region, temporal boundaries of a surgical complication, or spatial boundaries of the surgical complication; and
annotating the unlabeled real videos with the machine learning model by labeling the features of the unlabeled real videos identified by the machine learning model.
8. The non-transitory computer-readable medium of claim 7 , wherein the instructions, which when executed by the one or more processing units, cause the one or more processing units to perform further operations comprising:
providing, to a refiner neural network, simulated images from the simulated surgical videos before the pre-training; and
refining the simulated surgical videos with the refiner neural network, wherein the refiner neural network adjusts the simulated images until a discriminator neural network determines the simulated images are comparable to unlabeled real images within a first threshold, wherein the features annotated from the simulated surgical videos are included after the refining, and wherein the unlabeled real images are representative of the surgical scenario in the real environment.
9. The non-transitory computer-readable medium of claim 7 , wherein the context information for each of the simulated surgical videos includes at least one of three-dimensional spatial boundaries of the simulated surgical instrument, three-dimensional spatial boundaries of the simulated anatomical region, motion of the simulated surgical instrument, separation distance between the simulated surgical instrument and the simulated anatomical region, orientation of the simulated surgical instrument, temporal boundaries of one or more surgical steps of the simulated surgical task from the surgical scenario, temporal boundaries of a simulated surgical complication, or spatial boundaries of the simulated surgical complication.
10. The non-transitory computer-readable medium of claim 7 , wherein the instructions, which when executed by the one or more processing units, cause the one or more processing units to perform further operations comprising rendering markers or identifiers superimposed on top of a corresponding one of the unlabeled real videos to generate annotated surgical videos.
11. The non-transitory computer-readable medium of claim 7 , wherein the instructions, which when executed by the one or more processing units, cause the one or more processing units to perform further operations comprising displaying annotated surgical videos corresponding to the unlabeled real videos with the features annotated by the machine learning model superimposed on a corresponding one of the unlabeled real videos.
12. The non-transitory computer-readable medium of claim 7 , wherein the machine learning model includes at least one of a radial basis function neural network, a Kohonen self-organizing neural network, a recurrent neural network, a convolution neural network, or a modular neural network.
13. A surgical simulator for simulating a surgical scenario, the surgical simulator comprising:
a display system adapted to show simulated surgical videos to a user of the surgical simulator;
a user interface adapted to correlate a physical action of the user with a simulated action of the surgical simulator;
a controller including one or more processors coupled to memory, the display system, and the user interface, wherein the memory stores instructions that when executed by the one or more processors cause the surgical simulator to perform operations including:
generating the simulated surgical videos, each representative of a simulation of the surgical scenario;
associating simulated ground truth data from the simulation with the simulated surgical videos, wherein the simulated ground truth data corresponds to context information of at least one of a surgical instrument, a anatomical region, a surgical task, or a surgical action simulated by the surgical simulator;
annotating features of the simulated surgical videos based, at least in part, on the simulated ground truth data to generate annotated training data for training a machine learning model to probabilistically identify the features from at least one of unlabeled simulated videos or unlabeled real videos corresponding to the surgical scenario, wherein the features include at least one of a separation distance between the surgical instrument and the anatomical region, temporal boundaries of a surgical complication, or spatial boundaries of the surgical complication;
pre-training the machine learning model with the annotated training data;
receiving, from a surgical video database, real surgical videos representative of the surgical scenario, wherein the real surgical videos include the features annotated for the machine learning model; and
after the pre-training, training the machine learning model with the real surgical videos, wherein the training configures the machine learning model to probabilistically identify the features from the unlabeled real videos corresponding to the surgical scenario.