System for estimating image recognition information from at least images of energy device and biological tissue, computer-readable non-transitory information storage medium, and energy output adjustment method
The system includes a memory that stores first and second trained models, and a processor. The processor acquires a captured image in which at least one energy device and at least one biological tissue are imaged. The processor detects a bounding box from the captured image by processing based on the first trained model and estimates the image recognition information from the captured image in the bounding box by processing based on the second trained model. The processor outputs an energy output adjustment instruction based on the estimated image recognition information to the generator. The generator controls the energy supply amount to the energy device based on the energy output adjustment instruction.
1 . A system comprising:
a memory storing a trained model trained so as to output image recognition information from a training device tissue image or a training tissue image, the image recognition information being at least one of tissue information about at least one biological tissue or treatment information about treatment on the at least one biological tissue, the training device tissue image being an image of at least one energy device for performing energy output by receiving energy supply and the at least one biological tissue, the training tissue image being an image of the at least one biological tissue; and
a processor,
the processor being configured to perform:
acquiring a captured image that is an image of the at least one energy device and the at least one biological tissue;
executing processing based on the trained model stored in the memory to estimate the image recognition information from the captured image; and
outputting an energy output adjustment instruction based on the estimated image recognition information to a generator that controls an energy supply amount to the energy device based on the energy output adjustment instruction,
the trained model comprising:
a first trained model trained to detect a bounding box indicating a distal end section of the at least one energy device from the training device tissue image; and
a second trained model trained to output the image recognition information from the training device tissue image in the bounding box,
the processor, in the estimating, detecting the bounding box from the captured image by processing based on the first trained model and estimating the image recognition information from the captured image in the bounding box by processing based on the second trained model.
2 . The system as defined in claim 1 , wherein the processor performs:
determining any of adjustments of increasing, reducing, and maintaining the energy output from a reference energy output based on the image recognition information; and
outputting an instruction for the determined adjustment as the energy output adjustment instruction.
3 . The system as defined in claim 2 , wherein the processor performs
outputting the energy output adjustment instruction using a preset energy output or the energy output of the generator in real time as the reference energy output.
4 . The system as defined in claim 2 , wherein the processor performs:
acquiring an endoscope image from an endoscope as the captured image; and
outputting the energy output adjustment instruction using the energy output set at the time point when the endoscope image was acquired as the reference energy output.
5 . The system as defined in claim 1 , wherein
the energy device is a device that includes two jaws capable of gripping a tissue and that receives the energy supply from the generator and performs energy output from the two jaws.
6 . The system as defined in claim 1 , wherein
the tissue information includes tissue type or tissue condition of a tissue to be treated by the at least one energy device.
7 . The system as defined in claim 1 , wherein
the treatment information includes an amount of tissue gripped by the at least one energy device, or an amount of tissue traction by the at least one energy device or another device.
8 . The system as defined in claim 1 , wherein
the treatment information includes tension of a tissue treated by the at least one energy device, or a distance between the at least one energy device and an attention object.
9 . The system as defined in claim 1 , wherein
the processor changes priority of use of the image recognition information and electrical information obtained from the at least one energy device in controlling the energy output, based on estimation accuracy upon estimation of the image recognition information.
10 . The system as defined in claim 1 , wherein
the processor performs:
acquiring electrical information from the at least one energy device; and
changing priority of use of the image recognition information and the electrical information in controlling the energy output, based on consistency of the image recognition information and the electrical information.
11 . The system as defined in claim 1 , wherein
the processor performs
displaying the captured image and at least one of the content of the image recognition information and the content of the energy output adjustment instruction on a display.
12 . The system as defined in claim 1 , wherein
the trained model is trained to detect each biological tissue region of the at least one biological tissue from the training device tissue image or the training tissue image, and detect a distal end section region of the at least one energy device from the training device tissue image, and
the processor performs:
detecting the each biological tissue region and the distal end section region from the captured image by processing based on the trained model; and
estimating the image recognition information based on the detected each biological tissue region and the distal end section region.
13 . The system as defined in claim 1 , further comprising:
the at least one energy device; and
the generator.
14 . The system as defined in claim 13 , further comprising
an endoscope that captures an endoscope image as the captured image,
wherein the processor performs:
acquiring the endoscope image from the endoscope; and
estimating the image recognition information from the endoscope image by processing based on the trained model.
15 . A computer-readable non-transitory information storage medium storing a program for causing a computer to execute:
acquiring a captured image, which is an image of at least one energy device that receives energy supply and performs energy output and at least one biological tissue;
estimating image recognition information from the captured image by processing based on a trained model trained to output the image recognition information from a training device tissue image or a training tissue image, the image recognition information being at least one of tissue information about the at least one biological tissue or treatment information about treatment on the at least one biological tissue, the training device tissue image being an image of the at least one energy device and the at least one biological tissue, the training tissue image being an image of the at least one biological tissue;
outputting an energy output adjustment instruction based on the estimated image recognition information to a generator that controls an energy supply amount to the energy device based on the energy output adjustment instruction; and
in the estimating, detecting a bounding box from the captured image by processing based on a first trained model included in the trained model and estimating the image recognition information from the captured image in the bounding box by processing based on a second trained model included in the trained model, the first trained model being a model trained to detect the bounding box indicating a distal end section of the at least one energy device from the training device tissue image, the second trained model being a model trained to output the image recognition information from the training device tissue image in the bounding box.
16 . The computer-readable non-transitory information storage medium as defined in claim 15 , which stores a program for causing a computer to execute: determining any of adjustments of increasing, reducing, and maintaining the energy output from a reference energy output based on the image recognition information; and outputting an instruction for the determined adjustment as the energy output adjustment instruction.
17 . The computer-readable non-transitory information storage medium as defined in claim 16 , which stores a program for causing a computer to execute outputting the energy output adjustment instruction using a preset energy output or the energy output of the generator in real time as the reference energy output.
18 . An energy output adjustment method, comprising:
acquiring a captured image, which is an image of at least one energy device that receives energy supply and performs energy output and at least one biological tissue; and
estimating image recognition information from the captured image by processing based on a trained model trained to output the image recognition information from a training device tissue image or a training tissue image, the image recognition information being at least one of tissue information about the at least one biological tissue or treatment information about treatment on the at least one biological tissue, the training device tissue image being an image of the at least one energy device and the at least one biological tissue, the training tissue image being an image of the at least one biological tissue;
outputting an energy output adjustment instruction based on the estimated image recognition information to a generator that controls an energy supply amount to the energy device based on the energy output adjustment instruction; and
in the estimating, detecting a bounding box from the captured image by processing based on a first trained model included in the trained model and estimating the image recognition information from the captured image in the bounding box by processing based on a second trained model included in the trained model, the first trained model being a model trained to detect the bounding box indicating a distal end section of the at least one energy device from the training device tissue image, the second trained model being a model trained to output the image recognition information from the training device tissue image in the bounding box.
19 . The energy output adjustment method as defined in claim 18 , further comprising:
determining any of adjustments of increasing, reducing, and maintaining the energy output from a reference energy output based on the image recognition information; and
outputting an instruction for the determined adjustment as the energy output adjustment instruction.
20 . The energy output adjustment method as defined in claim 19 , further comprising
outputting the energy output adjustment instruction using a preset energy output or the energy output of the generator in real time as the reference energy output.