Operation supporting device and method of operating operation supporting device

1. An operation supporting device comprising a processor configured to:

acquire pre-operation data, the pre-operation data including position information of a feature portion from an inside of a body of a subject, is the position information being generated before an operation;

acquire a planned procedure in the operation that includes creation and simulation of an operation plan before the operation;

acquire an endoscope image generated in the operation;

generate real-time data including real-time position information of the feature portion based at least on the pre-operation data and the endoscope image;

generate an image to superimpose on the endoscope image based on the real-time data; and

recognize a real-time scene based at least on the planned procedure and the real-time data.

2. The operation supporting device according to claim 1 , wherein the processor is configured to:

acquire, as the pre-operation data, 3D volume data generated from information on the subject before the operation; and

recognize the feature portion in the 3D volume data and thereby configured to acquire the position information of the feature portion.

3. The operation supporting device according to claim 2 , wherein

the processor is configured to generate real-time space information about an inside of the body based on the endoscope image, and

based on the endoscope image and the real-time space information, the processor is configured to:

convert the 3D volume data acquired as the pre-operation data, and

generate real-time 3D volume data including the real-time position information of the feature portion.

4. The operation supporting device according to claim 1 , wherein

the processor is configured to acquire, as the pre-operation data, the planned procedure and an estimated required time for each of a plurality of tasks of the planned procedure.

5. The operation supporting device according to claim 1 , wherein the processor is configured to:

recognize a bleeding position when bleeding occurs in the endoscope image, and

generate a navigation image relating to the recognized bleeding position to superimpose on the endoscope image.

6. The operation supporting device according to claim 1 , wherein:

the processor is configured to acquire information sensed, measured, or observed by a device disposed outside the body in the operation, and

the processor is configured to recognize the real-time scene based on the planned procedure and the real-time data and the information sensed, measured, or observed by the device disposed outside the body.

7. The operation supporting device according to claim 6 , wherein:

the processor is configured to acquire, as the information sensed, measured, or observed by the device disposed outside the body, a position of a port through which a surgical instrument including an endoscope and a treatment instrument is inserted, a type of surgical instrument inserted from the port, and information about an operator of the operation.

8. The operation supporting device according to claim 1 , wherein:

the processor is configured to recognize a real-time task from among a plurality of tasks of the planned procedure based on the recognized real-time scene, and

the processor is configured to display the planned procedure with the endoscope image such that the recognized real-time task is highlighted in the displayed procedure.

9. The operation supporting device according to claim 1 , wherein:

the processor is configured to select and generate information to be displayed together with the endoscope image, based on the recognized real-time scene.

10. The operation supporting device according to claim 1 , wherein:

The processor is configured to estimate a probability that risk will be incurred in real time, based on the recognized real-time scene, and

the processor is configured to issue a warning when an estimated probability is higher than a predetermined value.

11. The operation supporting device according to claim 10 , wherein:

the processor is configured to estimate the probability that risk will be incurred by determining

a probability of causing bleeding or damage to a nerve,

a probability of interference between surgical instruments including an endoscope and a treatment instrument, and

a probability of interference of the surgical instruments with an organ.

12. The operation supporting device according to claim 1 , wherein:

the processor is configured to cause recommended information relating to treatment in the recognized real-time scene to be displayed together with the endoscope image, based on the recognized real-time scene.

13. The operation supporting device according to claim 12 , wherein:

the processor is configured to cause one or more of the following to be displayed as the recommended information relating to the treatment in the recognized scene: (1) treatment information in a scene similar to the recognized real-time scene, (2) information on an accidental symptom relating to the recognized real-time scene, (3) a type of a treatment instrument recommended in the recognized real-time scene, and (4) recommended setting values for surgical instruments including an endoscope and the treatment instrument in the recognized real-time scene.

14. The operation supporting device according to claim 1 , wherein:

the processor is configured to acquire, as the pre-operation data: (1) 3D volume data generated from information on the subject before the operation, and (2) a result of a simulation of the operation performed before the operation.

15. The operation supporting device according to claim 14 , wherein:

the processor is configured to make a judgment as to a necessity for re-execution of the simulation of the operation, based on the recognized real-time scene and the result of the simulation of the operation acquired as the pre-operation data.

16. The operation supporting device according to claim 15 , wherein:

the processor is configured to generate real-time 3D volume data including the real-time position information of the feature portion, based at least on the pre-operation data and the endoscope image, and

when it is judged to re-execute the simulation of the operation, the processor is configured to re-execute simulation of the operation in the real-time 3D volume data, and configured to acquire a result of the re-executed simulation of the operation.

17. The operation supporting device according to claim 1 , wherein the processor is configured to:

acquire diagnostic information and patient information before the operation;

generate 3D volume data of an inside of the body of the subject based at least on the diagnostic information and the patient information;

set the position information of the feature portion in the generated 3D volume data;

perform simulation of the operation in the generated 3D volume data; and

create the planned procedure based on a result of the simulation.

18. The operation supporting device according to claim 17 , wherein

the processor is configured to:

acquire an operation plan including the planned procedure based on the result of the simulation,

recognize the feature portion on the acquired endoscope image,

generate real-time 3D volume data of an inside of the body of the subject based at least on the acquired endoscope image and the 3D volume data generated before the operation, and

recognize the real-time scene based at least on the real-time 3D volume data and the operation plan.

19. A method of operating an operation supporting device, the method comprising:

acquiring a planned procedure in an operation that includes creation and simulation of an operation plan before the operation;

acquiring an endoscope image generated in the operation;

recognizing a feature portion of a subject on the acquired endoscope image;

generating real-time 3D volume data of an inside of a body of the subject including the feature portion, based on the acquired endoscope image; and

recognizing a real-time scene based at least on the real-time 3D volume data and the planned procedure.

20. The operation supporting according to claim 1 , wherein the feature portion includes at least one of the following: a vessel blood vessel, a lymph vessel, an organ, a nerve, a tumor, or a lymph node.