Ultrasound imaging plane alignment guidance for neural networks and associated devices, systems, and methods
Ultrasound image devices, systems, and methods are provided. In one embodiment, a guidance system for obtaining a medical image includes a processor configured to obtain a motion control configuration for repositioning an imaging device from a first imaging position to a second imaging position with respect to a subject's body, the motion control configuration obtained based on a predictive network, an image of the subject's body captured while the imaging device is positioned at the first imaging position, and a target image view including a clinical property; and a display in communication with the processor and configured to display an instruction, based on the motion control configuration, for operating a control component in communication with the imaging device such that the imaging device is repositioned to the second imaging position.
1 . A guidance system for obtaining a medical image, comprising:
a processor configured to:
obtain a motion control configuration for repositioning an imaging device configured to be inserted inside a subject's body from a first imaging position to a second imaging position with respect to the subject's body using a control component in communication with the imaging device, wherein the motion control configuration is obtained based on:
(i) a predictive network comprising at least a first convolutional neural network trained using one or more reference images,
(ii) an image of the subject's body captured while the imaging device is positioned at the first imaging position, and
(iii) a target image view including a clinical property, wherein the motion control configuration is obtained based on the predictive network without comparison to a reference image, wherein the first convolutional neural network is configured to compute a probability of reaching the target image view for each of a plurality of candidate motion control configurations and select the candidate motion control configuration with a highest probability of reaching the target image view; and wherein the motion control configuration corresponds to a motion vector generated using the predictive network based on the image of the subject's body captured while the imaging device is positioned at the first imaging position and the target image view including a clinical property;
determine, via at least a second convolutional neural network, whether an image captured while the imaging device is positioned at the second imaging position contains the target image view, wherein the second convolutional neural network is trained to qualify whether an input image includes a target image view or a predetermined or selected clinical property;
if the image captured while the imaging device is positioned at the second imaging position does not contain the target image view, repeat the obtaining operation to generate a new motion control configuration for repositioning the imaging device to a new imaging position using at least the first convolutional neural network; and
if the image captured while the imaging device is positioned at the second imaging position does contain the target image view:
generating a plurality of modified motion vectors based on the motion vector generated using the predictive network;
repositioning the imaging device based on each of the modified motion vectors;
capturing an image of the target image view including the clinical property at each of the modified motion vectors; and
applying at least a third convolutional neural network to the captured images of the target image view to select an image including the target image view having the highest quality from among the captured images, wherein the third convolutional neural network is trained to compare pairs of input images and select an input image having a higher quality level from the pair of input images;
a display in communication with the processor and configured to display an instruction, based on at least one of the motion control configuration and the new motion control configuration, for operating the control component such that the imaging device is repositioned to the second imaging position when the instruction is based on the motion control configuration displayed, or the new imaging position when the instruction is based on the new motion control configuration.
2 . The guidance system of claim 1 , wherein the display is further configured to display the instruction by displaying a graphical view of the control component and a visual indicator indicating at least one of a direction of a movement for operating the control component or an amount of the movement.
3 . The guidance system of claim 2 , wherein the graphical view includes a perspective view of the control component.
4 . The guidance system of claim 2 , further comprising a communication device in communication with the processor and configured to receive a request for a first view of a plurality of views of the control component, wherein the processor is further configured to switch the graphical view from a current view of the plurality of views to the first view based on the request.
5 . The guidance system of claim 4 , wherein the plurality of views include at least one of an anterior view of the control component, a sideview of the control component, or a posterior view of the control component.
6 . The guidance system of claim 1 , wherein the control component includes at least one of a first sub-component that controls a movement of the imaging device within the subject's body and along a left-right plane of the subject's body, a second sub component that controls a movement of the imaging device within the subject's body and along anterior-posterior plane of the subject's body, or a third sub-component that controls an orientation of an imaging plane of the imaging device within the subject's body.
7 . The guidance system of claim 6 , wherein the motion control configuration includes at least one of a first parameter for operating the first sub-component, a second parameter for operating the second sub-component, a third parameter for operating the third sub-component, or a fourth parameter for rotating the imaging device with respect to an axis of the imaging device.
8 . The guidance system of claim 7 , wherein the display is further configured to display the instruction by displaying a visual indicator including at least one of an arrow or an on/off indicator based on at least one of the first parameter, the second parameter, the third parameter, or the fourth parameter.
9 . The guidance system of claim 1 , wherein the imaging device is a transesophageal echocardiography (TEE) probe.
10 . The guidance system of claim 9 , wherein the control component is disposed on a handle of the TEE probe.
11 . The guidance system of claim 1 , wherein the first convolutional neural network of the predictive network is trained by:
providing a plurality of images obtained by the imaging device from at least two imaging positions to obtain the target image view;
obtaining a plurality of motion control configurations based an orientation or a movement of the imaging device associated with the at least two imaging positions; and
assigning a score to a relationship between the plurality of motion control configurations and the plurality of images with respect to the target image view.
12 . The guidance system of claim 1 , further comprising a communication device in communication with the processor and configured to receive the image, wherein the processor is further configured to obtain the motion control configuration by applying the predictive network to the image and a plurality of parameters for operating the control component.
13 . The guidance system of claim 12 , wherein the communication device is further configured to receive the plurality of parameters.
14 . A method of providing medical ultrasound imaging guidance, comprising:
obtaining a motion control configuration for repositioning an imaging device configured to be inserted inside a subject's body from a first imaging position to a second imaging position with respect to a subject's body using a control component in communication with the imaging device, wherein the motion control configuration is obtained based on:
(i) a predictive network comprising at least a first convolutional neural network trained using one or more reference images,
(ii) an image of the subject's body captured while the imaging device is positioned at the first imaging position, and
(iii) a target image view including a clinical property, wherein the motion control configuration is obtained based on the predictive network without comparison to a reference image, wherein the first convolutional neural network is configured to compute a probability of reaching the target image view for each of a plurality of candidate motion control configurations and select the candidate motion control configuration with a highest probability of reaching the target image view; and wherein the motion control configuration corresponds to a motion vector generated using the predictive network based on the image of the subject's body captured while the imaging device is positioned at the first imaging position and the target image view including a clinical property;
displaying an instruction, based on the motion control configuration, on a display for operating the control component such that the imaging device is repositioned to the second imaging position;
determining, via at least a second convolutional neural network, whether an image captured while the imaging device is positioned at the second imaging position contains the target image view, wherein the second convolutional neural network is trained to qualify whether an input image includes a target image view or a predetermined or selected clinical property;
if the image captured while the imaging device is positioned at the second imaging position does not contain the target image view, repeating the obtaining step to generate a new motion control configuration for repositioning the imaging device to a new imaging position using at least the first convolutional neural network and displaying an instruction, based on the new motion control configuration, on the display for operating the control component such that the imaging device is repositioned to the new imaging position; and
if the image captured while the imaging device is positioned at the second imaging position does contain the target image view:
generating a plurality of modified motion vectors based on the motion vector generated using the predictive network;
repositioning the imaging device based on each of the modified motion vectors;
capturing an image of the target image view including the clinical property at each of the modified motion vectors; and
applying at least a third convolutional neural network to the captured images of the target image view to select an image including the target image view having the highest quality from among the captured images, wherein the third convolutional neural network is trained to compare pairs of input images and select an input image having a higher quality level from the pair of input images.
15 . The method of claim 14 , wherein the displaying includes displaying a graphical view of the control component and a visual indicator indicating at least one of a direction of a movement for operating the control component or an amount of the movement.
16 . The method of claim 15 , further comprising:
receiving a request for a first view of a plurality of views of the control component; and
switching the graphical view from a current view of the plurality of views to the first view based on the request.
17 . The method of claim 16 , wherein the plurality of views include at least one of an anterior view of the control component, a sideview of the control component, or a posterior view of the control component.
18 . The method of claim 14 , wherein the control component includes at least one of a first sub-component that controls a movement of the imaging device within the subject's body and along a left-right plane of the subject's body, a second sub component that controls a movement of the imaging device within the subject's body and along anterior-posterior plane of the subject's body, or a third sub-component that controls an orientation of an imaging plane of the imaging device within the subject's body.
19 . The method of claim 18 , wherein the motion control configuration includes at least one of a first parameter for operating the first sub-component, a second parameter for operating the second sub-component, a third parameter for operating the third sub-component, or a fourth parameter for rotating the imaging device with respect to an axis of the imaging device.
20 . The method of claim 18 , wherein the displaying includes displaying a visual indicator including at least one of an arrow or an on/off indicator based on the at least one of a first parameter, a second parameter, or a third parameter.