Machine learning system for identifying a state of a surgery, and assistance function

1. A computer-implemented method when executed by data processing hardware causes the data processing hardware to perform operations comprising:

capturing digital image data of a surgical microscope, the digital image data generated during an ophthalmological surgery by an image sensor and annotated;

capturing sensor data of a phaco system, the sensor data generated during the ophthalmological surgery by a sensor of the phaco system and annotated;

wherein the annotated sensor data and the annotated digital image data have synchronized timestamps, and wherein the annotations refer in indicative fashion to a state of an ophthalmological surgery, and

training a machine learning system using the annotated image data and the annotated sensor data to generate a learning model to predict a state of an ophthalmological surgery and, on the basis thereof, output a control signal,

wherein the control signal is suitable for facilitating an assistance function in a subsequent use of the trained learning model during a prediction phase of the machine learning system.

2. The method of claim 1 , further comprising:

capturing sensor data and/or image data of a surgery robot, the sensor data and/or image data of the surgery robot generated during the ophthalmological surgery and annotated; and

using the annotated sensor data and/or image data of the surgery robot during the training of the machine learning system to generate the learning model.

3. The method of claim 1 , further comprising:

capturing optical coherence tomography (OCT) image data of an OCT scanner and/or of a surgery robot, the OCT image data and/or the axis position data of the surgery robot generated during the ophthalmological surgery and annotated; and

using the annotated OCT image data and/or the axis position data during the training of the machine learning system to generate the learning model.

4. The method of claim 1 , wherein the predicted the state of the ophthalmological surgery is based on a determination of a surgical instrument characteristic for a respective phase of surgery by applying the machine learning system.

5. The method of claim 1 , wherein the control signal is able to be used to control at least one parameter of devices that are used during the ophthalmological surgery.

6. The method of claim 1 , wherein the digital image data and the sensor data are captured in time-synchronized fashion in a joint storage system.

7. A method for using a learning model for predicting a state of an ophthalmological surgery, the method comprising:

capturing and annotating digital image data of a surgical microscope during the ophthalmological surgery;

capturing and annotating sensor data of a phaco system during the ophthalmological surgery, the annotated sensor data and the annotated digital image data have synchronized timestamps, and the annotations referring in indicative fashion to a state of an ophthalmological surgery;

training a machine learning system using the annotated digital image data and the annotated sensor data to generate and train a learning model to predict a state of an ophthalmological surgery and, on the basis thereof, output a control signal, the control signal suitable for facilitating an assistance function in a subsequent use of the trained learning model during a prediction phase of the machine learning system;

after training the machine learning system, determining a state of the ophthalmological surgery using the trained machine learning system that has the trained learning model adapted to:

predict an ophthalmological surgery state; and

on the basis of the predicted ophthalmological surgery state, output a control signal; and

controlling an assistance function by the control signal during a prediction phase of the machine learning system during the ophthalmological surgery.

8. The method of claim 7 , further comprising:

capturing sensor data and/or image data of a surgery robot, the sensor data and/or image data of the surgery robot generated during the ophthalmological surgery; and

using the sensor data and/or image data of the surgery robot as input for the trained learning model.

9. The method of claim 7 , further comprising:

capturing optical coherence tomography (OCT) image data of an OCT scanner and/or of a surgery robot, the OCT image data and/or the axis position data of the surgery robot generated during the ophthalmological surgery; and

using the OCT image data and/or the axis position data of the surgery robot as input for the trained learning model.

10. The method of claim 7 , wherein determining the state of the ophthalmological surgery is based on a determination of a surgical instrument characteristic for a respective phase of surgery by applying the machine learning system.

11. The method of claim 7 , wherein the control signal is able to be used to control at least one parameter of devices that are used during the ophthalmological surgery.

12. The method of claim 7 , wherein the digital image data and the sensor data are captured in time-synchronized fashion in a joint storage system.

13. A surgery assistance system for using a learning model for predicting a state of an ophthalmological surgery and, on the basis thereof, outputting a control signal, wherein the surgery assistance system comprises:

a first capture module configured to capture digital image data of a surgical microscope during an ophthalmological surgery, wherein the first capture module and the surgical microscope are electrically interconnected for signal interchange;

a second capture module configured to capture sensor data of a phaco system during the ophthalmological surgery, wherein the second capture module and the phaco system are electrically connected to a data processing system for signal interchange;

a prediction module of a machine learning system, the prediction module having a trained learning model adapted by preceding training to predict a state of the ophthalmological surgery and, on the basis of the predicted state of the ophthalmological surgery, output a control signal; and

a control module configured to:

receive the control signal as an input signal;

during a prediction phase of the machine learning system during the ophthalmological surgery, control a parameter of a device used during the ophthalmological surgery;

capture digital image data of a surgical microscope, the digital image data generated during an ophthalmological surgery by an image sensor and annotated;

capture sensor data of a phaco system, the sensor data generated during the ophthalmological surgery by a sensor of the phaco system and annotated;

wherein the annotated sensor data and the annotated digital image data have synchronized timestamps, and wherein the annotations refer in indicative fashion to a state of an ophthalmological surgery, and

train a machine learning system using the annotated image data and the annotated sensor data to generate a learning model to predict a state of an ophthalmological surgery and, on the basis thereof, output a control signal,

wherein the control signal is suitable for facilitating an assistance function in a subsequent use of the trained learning model during a prediction phase of the machine learning system.

14. The surgery assistance system of claim 13 , wherein the preceding training of the learning model comprises training the learning model using the digital image data and/or the sensor data.

15. The surgery assistance system of claim 13 , wherein the digital image data and the sensor data are captured in time-synchronized fashion.

16. The surgery assistance system of claim 13 , wherein the predicted state of the ophthalmological surgery is based on a determination of a surgical instrument characteristic for a respective phase of surgery by applying the machine learning system.