Augmented reality interface for assisting a user to operate an ultrasound device
Aspects of the technology described herein relate to techniques for guiding an operator to use an ultrasound device. Thereby, operators with little or no experience operating ultrasound devices may capture medically relevant ultrasound images and/or interpret the contents of the obtained ultrasound images. For example, some of the techniques disclosed herein may be used to identify a particular anatomical view of a subject to image with an ultrasound device, guide an operator of the ultrasound device to capture an ultrasound image of the subject that contains the particular anatomical view, and/or analyze the captured ultrasound image to identify medical information about the subject.
1. A method, comprising:
capturing, using a camera coupled to a mobile device, an image of an ultrasound device being used by an operator when the ultrasound device is in the camera's field of view;
identifying a pose of the ultrasound device in the image of the ultrasound device;
generating, using at least one processor of the mobile device, a composite image at least in part by overlaying, onto the image of the ultrasound device and using the pose of the ultrasound device, an indication of a direction in which the operator is to move the ultrasound device, wherein overlaying the indication onto the image of the ultrasound device comprises overlaying an arrow onto at least part of the ultrasound device in the image of the ultrasound device, the arrow indicating the direction in which the operator is to move the ultrasound device; and
presenting the composite image to the operator via a display of the mobile device,
wherein the method further comprises:
prior to generating the composite image, using the mobile device to:
analyze an ultrasound image captured by the ultrasound device by using a multi-layer neural network, and
determine the indication using an output of the multi-layer neural network.
2. The method of claim 1 , wherein the ultrasound device has a marker disposed thereon, and wherein capturing the image of the ultrasound device comprises obtaining an image of the marker.
3. The method of claim 2 , wherein identifying the pose of the ultrasound device comprises identifying a location of the marker in the image of the ultrasound device.
4. The method of claim 1 , further comprising:
obtaining the ultrasound image captured by the ultrasound device.
5. The method of claim 4 , wherein the method further comprises presenting the ultrasound image with the composite image via the display of the mobile device.
6. The method of claim 1 , wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to rotate the ultrasound device.
7. The method of claim 1 , wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to reposition the ultrasound device.
8. The method of claim 1 , wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to move the ultrasound device up, down, left, or right.
9. A system, comprising:
a mobile device comprising a display and at least one processor configured to:
capture, using a camera coupled to the mobile device, an image of an ultrasound device being used by an operator when the ultrasound device is in the camera's field of view;
identify a pose of the ultrasound device in the image of the ultrasound device;
generate, using the at least one processor of the mobile device, a composite image at least in part by overlaying, onto the image of the ultrasound device and using the pose of the ultrasound device, an indication of a direction in which the operator is to move the ultrasound device, wherein overlaying the indication onto the image of the ultrasound device comprises overlaying an arrow onto at least part of the ultrasound device in the image of the ultrasound device, the arrow indicating the direction in which the operator is to move the ultrasound device; and
cause the display to present the composite image to the operator,
wherein the at least one processor of the mobile device is further configured to:
prior to generating the composite image:
analyze an ultrasound image captured by the ultrasound device by using a multi-layer neural network, and
determine the indication using an output of the multi-layer neural network.
10. The system of claim 9 , wherein the ultrasound device comprises a marker disposed thereon, wherein the image of the ultrasound device comprises an image of the marker, and wherein the at least one processor is configured to identify the pose of the ultrasound device at least in part by identifying a location of the marker in the image of the ultrasound device.
11. The system of claim 10 , wherein the marker is selected from the group consisting of: a holographic marker, a dispersive marker, and an ArUco marker.
12. The system of claim 9 , further comprising:
the ultrasound device.
13. The system of claim 9 , wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to rotate the ultrasound device.
14. The system of claim 9 , wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to reposition the ultrasound device.
15. The system of claim 9 , wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to move the ultrasound device up, down, left, or right.
16. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one processor of a mobile device, cause the at least one processor to:
capture, using a camera coupled to the mobile device, an image of an ultrasound device being used by an operator when the ultrasound device is in the camera's field of view;
identify a pose of the ultrasound device in the image of the ultrasound device;
generate, using the at least one processor of the mobile device, a composite image at least in part by overlaying, onto the image of the ultrasound device and using the pose of the ultrasound device, an indication of a direction in which the operator is to move the ultrasound device, wherein overlaying the indication onto the image of the ultrasound device comprises overlaying an arrow onto at least part of the ultrasound device in the image of the ultrasound device, the arrow indicating the direction in which the operator is to move the ultrasound device; and
cause the display to present the composite image to the operator,
wherein the processor-executable instructions further cause the at least one processor of the mobile device to:
prior to generating the composite image:
analyze an ultrasound image captured by the ultrasound device by using a multi-layer neural network, and
determine the indication using an output of the multi-layer neural network.
17. The at least one non-transitory computer-readable storage medium of claim 16 , wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to rotate the ultrasound device.
18. The at least one non-transitory computer-readable storage medium of claim 16 , wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to reposition the ultrasound device.
19. The at least one non-transitory computer-readable storage medium of claim 16 , wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to move the ultrasound device up, down, left, or right.