Systems and methods for guiding an ultrasound probe
An ultrasound device ( 10 ) includes a probe ( 12 ) including a tube ( 14 ) sized for insertion into a patient and an ultrasound transducer ( 18 ) disposed at a distal end ( 16 ) of the tube. A camera ( 20 ) is mounted at the distal end of the tube in a fixed spatial relationship to the ultrasound transducer. At least one electronic processor ( 28 ) is programmed to: control the ultrasound transducer and the camera to acquire ultrasound images ( 19 ) and camera images ( 21 ) respectively while the ultrasound transducer is disposed in vivo inside the patient; and construct a keyframe ( 36 ) representative of an in vivo position of the ultrasound transducer including at least ultrasound image features ( 38 ) extracted from at least one of the ultrasound images acquired at the in vivo position of the ultrasound transducer and camera image features ( 40 ) extracted from one of the camera images acquired at the in vivo position of the ultrasound transducer.
1. An ultrasound device, comprising:
a probe including a tube sized for insertion into a patient and an ultrasound transducer disposed at a distal end of the tube;
a camera mounted at the distal end of the tube in a fixed spatial relationship to the ultrasound transducer;
at least one electronic processor programmed to:
control the ultrasound transducer and the camera to acquire ultrasound images and camera images respectively while the ultrasound transducer is disposed in vivo inside the patient; and
construct a keyframe representative of an in vivo position of the ultrasound transducer, wherein the keyframe is a single image and wherein the keyframe includes at least: (i) ultrasound image features extracted from at least one of the ultrasound images acquired at the in vivo position of the ultrasound transducer, wherein the keyframe further includes, displayed within the single image, one or more settings of the ultrasound transducer at the acquisition time of the ultrasound image acquired at the in vivo position of the ultrasound transducer; and (ii) camera image features extracted from one of the camera images acquired at the in vivo position of the ultrasound transducer; and
a display configured to display the keyframe.
2. The ultrasound device of claim 1 , wherein the keyframe further includes rotation settings, insertion settings, and joint position settings of the probe at the acquisition time of the ultrasound image acquired at the in vivo position of the ultrasound transducer.
3. The ultrasound device of claim 1 , wherein the ultrasound transducer and the camera are attached to each other or housed in or secured to a common rigid housing disposed at the distal end of the tube.
4. The ultrasound device of claim 3 , wherein the ultrasound transducer is arranged at the distal end of the tube to be side-emitting, and the camera is arranged at the distal end of the tube to be forward-facing.
5. The ultrasound device of claim 1 , wherein the at least one electronic processor is programmed to construct the keyframe responsive to satisfaction of a keyframe acquisition criterion.
6. The ultrasound device of claim 5 , wherein the keyframe acquisition criterion comprises a comparison between a last keyframe and currently acquired ultrasound and camera images.
7. The ultrasound device of claim 5 , wherein the keyframe acquisition criterion comprises a recognition of a defining image feature of a target tissue imaged in a current ultrasound image.
8. The ultrasound device of claim 1 , further including at least one user input device; and wherein the at least one electronic processor is programmed to:
label the keyframe representative of the in vivo position of the ultrasound transducer upon receiving a user input via the at least one user input device.
9. The ultrasound device of claim 1 , wherein the at least one electronic processor is further programmed to:
construct a keyframe representative of a first view consisting of a first in vivo position of the ultrasound transducer;
during a traversal of the ultrasound transducer from the first view to a second view consisting of a second in vivo position of the ultrasound transducer, construct keyframes representative of intermediate positions of the ultrasound transducer; and
at the end of the traversal, construct a keyframe representative of the second view.
10. The ultrasound device of claim 9 , wherein the at least one electronic processor is further programmed to:
during a backtracking traversal of the ultrasound transducer back from the second view to the first view or an intermediate view therebetween, provide human-perceptible guidance for manual control of the probe based on comparisons of ultrasound images and camera images acquired during backtracking traversal with the keyframes representative of the intermediate positions and the keyframe representative of the first view.
11. The ultrasound device of claim 10 , wherein the human-perceptible guidance includes commands including one or more advancement of the ultrasound device through the esophagus, retraction of the ultrasound device through the esophagus, and capture a keyframe.
12. The ultrasound device of claim 9 , wherein the at least one electronic processor is further programmed to:
control servomotors of the probe to perform the traversal of the ultrasound transducer from the first view to the second view; and
control the servomotors of the probe to perform a backtracking traversal of the ultrasound transducer back from the second view to the first view based on comparisons of ultrasound images and camera images acquired during the backtracking traversal with the keyframes representative of the intermediate positions and the keyframe representative of the first view.
13. The ultrasound device of claim 1 , wherein the probe comprises a transesophageal echocardiography (TEE) probe.
14. An ultrasound device, comprising:
a probe including a tube sized for insertion into a patient and an ultrasound transducer disposed at a distal end of the tube and arranged to be side-emitting;
a camera mounted at the distal end of the tube in a fixed spatial relationship to the ultrasound transducer and arranged to be forward-facing;
at least one electronic processor programmed to:
control the ultrasound transducer and the camera to acquire ultrasound images and camera images respectively while the ultrasound transducer is disposed in vivo inside the patient; and
construct a keyframe representative of an in vivo position of the ultrasound transducer, wherein the keyframe is a single image and wherein the keyframe includes at least: (i) ultrasound image features extracted from at least one of the ultrasound images acquired at the in vivo position of the ultrasound transducer, further including one or more settings of the ultrasound transducer at the acquisition time of the ultrasound image acquired at the in vivo position of the ultrasound transducer; and (ii) camera image features extracted from one of the camera images acquired at the in vivo position of the ultrasound transducer; and
a display configured to display the keyframe.
15. The ultrasound device of claim 14 , wherein the keyframe further includes one or more of:
one or more settings of the ultrasound transducer, rotation settings of the probe, insertion settings of the probe, and joint position settings of the probe at the acquisition time of the ultrasound image acquired at the in vivo position of the ultrasound transducer.
16. The ultrasound device of claim 14 , wherein the at least one electronic processor is further programmed to:
construct a keyframe representative of a first view consisting of a first in vivo position of the ultrasound transducer;
during a traversal of the ultrasound transducer from the first view to a second view consisting of a second in vivo position of the ultrasound transducer, construct keyframes representative of intermediate positions of the ultrasound transducer; and
at the end of the traversal, construct a keyframe representative the second view.
17. The ultrasound device of claim 16 , wherein the at least one electronic processor is further programmed to:
during a backtracking traversal of the ultrasound transducer back from the second view to the first view, provide human-perceptible guidance for manual control of the probe based on comparisons of ultrasound images and camera images acquired during backtracking traversal with the keyframes representative of the intermediate positions and the keyframe representative of the first view.
18. The ultrasound device of claim 16 , wherein the at least one electronic processor is further programmed to:
control servomotors of the probe to perform the traversal of the ultrasound transducer from the first view to the second view; and
control the servomotors of the probe to perform a backtracking traversal of the ultrasound transducer back from the second view to the first view based on comparisons of ultrasound images and camera images acquired during the backtracking traversal with the keyframes representative of the intermediate positions and the keyframe representative of the first view.
19. A method of controlling an ultrasound device comprising a probe including a tube sized for insertion into a patient and an ultrasound transducer disposed at a distal end of the tube and a camera mounted at the distal end of the tube in a fixed spatial relationship to the ultrasound transducer, the method comprising:
controlling the ultrasound transducer and the camera to acquire ultrasound images and camera images respectively while the ultrasound transducer is disposed in vivo inside the patient;
constructing a keyframe representative of a first view consisting of a first in vivo position of the ultrasound transducer, wherein the keyframe is a single image and wherein the keyframe includes including at least: (i) the ultrasound images, (ii) the camera images, (iii) ultrasound image features extracted from at least one of the ultrasound images acquired at the in vivo position of the ultrasound transducer further including one or more settings of the ultrasound transducer at the acquisition time of the ultrasound image acquired at the in vivo position of the ultrasound transducer; and (iv) camera image features extracted from one of the camera images acquired at the in vivo position of the ultrasound transducer;
during a traversal of the ultrasound transducer from the first view to a second view consisting of a second in vivo position of the ultrasound transducer, constructing keyframes representative of intermediate positions of the ultrasound transducer; and
at the end of the traversal, constructing a keyframe representative the second view.