Augmented reality interface for assisting a user to operate an ultrasound device
Aspects of the technology described herein relate to techniques for guiding an operator to use an ultrasound device. Thereby, operators with little or no experience operating ultrasound devices may capture medically relevant ultrasound images and/or interpret the contents of the obtained ultrasound images. For example, some of the techniques disclosed herein may be used to identify a particular anatomical view of a subject to image with an ultrasound device, guide an operator of the ultrasound device to capture an ultrasound image of the subject that contains the particular anatomical view, and/or analyze the captured ultrasound image to identify medical information about the subject.
1. A method for guiding an operator of an ultrasound device in positioning the ultrasound device, the method comprising:
using a single mobile device, separate from the ultrasound device and comprising at least one processor, a camera, and a display, to perform:
automatically identifying a pose of the ultrasound device at least in part by imaging the ultrasound device with the camera of the single mobile device;
generating, using the single mobile device, a composite image of the ultrasound device and an indication for how to move the ultrasound device, the generating comprising overlaying the indication on an image of the ultrasound device at a location in the image determined using the identified pose of the ultrasound device; and
presenting the composite image to the operator of the ultrasound device using the display of the single mobile device.
2. The method of claim 1 , wherein automatically identifying the pose of the ultrasound device comprises identifying a location, in the image, of a marker on the ultrasound device.
3. The method of claim 2 , wherein automatically identifying the pose of the ultrasound device comprises identifying a position of the ultrasound device in the image using the identified location of the marker in the image.
4. The method of claim 1 , wherein the ultrasound device comprises at least one sensor configured to detect movement of the ultrasound device, and wherein automatically identifying the pose of the ultrasound device is performed using data collected by the at least one sensor.
5. The method of claim 4 , wherein the at least one sensor comprises a gyroscope.
6. The method of claim 4 , wherein the at least one sensor comprises an accelerometer.
7. The method of claim 1 , further comprising:
obtaining an ultrasound image captured by the ultrasound device; and
generating the indication using the ultrasound image.
8. The method of claim 7 , further comprising:
presenting the ultrasound image to the operator of the ultrasound device using the display of the mobile device.
9. A system for guiding an operator of an ultrasound device in positioning the ultrasound device, the system comprising:
a single mobile device, separate from the ultrasound device and comprising at least one processor, a camera, and a display, the single mobile device configured to:
automatically identify a pose of the ultrasound device at least in part by imaging the ultrasound device with the camera of the single mobile device;
generate, using the single mobile device, a composite image of the ultrasound device and an indication for how to move the ultrasound device, the generating comprising overlaying the indication on an image of the ultrasound device at a location in the image determined using the identified pose of the ultrasound device; and
present the composite image to the operator of the ultrasound device using the display of the single mobile device.
10. The system of claim 9 , further comprising:
the ultrasound device.
11. The system of claim 10 , wherein the ultrasound device comprises at least one sensor configured to detect movement of the ultrasound device.
12. The system of claim 9 , wherein the single mobile device is configured to automatically identify the pose of the ultrasound device at least in part by identifying a location, in the image, of a marker on the ultrasound device.
13. The system of claim 12 , wherein the single mobile device is configured to automatically identify the pose of the ultrasound device at least in part by identifying a position of the ultrasound device in the image using the identified location of the marker in the image.
14. The system of claim 11 , wherein the single mobile device is configured to automatically identify the pose of the ultrasound device at least in part by using data collected by the at least one sensor.
15. The system of claim 14 , wherein the at least one sensor comprises a gyroscope.
16. The system of claim 14 , wherein the at least one sensor comprises an accelerometer.
17. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by a single mobile device, the single mobile device comprising at least one processor, a camera, and a display, cause the single mobile device to:
automatically identify a pose of an ultrasound device at least in part by imaging the ultrasound device with the camera of the single mobile device;
generate, using the single mobile device, a composite image of the ultrasound device and an indication for how to move the ultrasound device, the generating comprising overlaying the indication on an image of the ultrasound device at a location in the image determined using the identified pose of the ultrasound device; and
present the composite image to an operator of the ultrasound device using the display of the single mobile device.
18. The method of claim 1 , wherein overlaying the indication on the image of the ultrasound device comprises overlaying an indication to move the ultrasound device in a specific direction.
19. The method of claim 18 , wherein overlaying the indication to move the ultrasound device in the specific direction comprises overlaying, on the image of the ultrasound device, an arrow pointing in the direction.
20. The method of claim 1 , wherein overlaying the indication on the image of the ultrasound device comprises overlaying an indication to rotate the ultrasound device.
21. The system of claim 9 , wherein overlaying the indication on the image of the ultrasound device comprises overlaying an indication to move the ultrasound device in a specific direction.
22. The system of claim 9 , wherein overlaying the indication on the image of the ultrasound device comprises overlaying an indication to rotate the ultrasound device.
23. The at least one non-transitory computer-readable storage medium of claim 17 , wherein overlaying the indication on the image of the ultrasound device comprises overlaying an indication to move the ultrasound device in a specific direction.
24. The at least one non-transitory computer-readable storage medium of claim 17 , wherein overlaying the indication on the image of the ultrasound device comprises overlaying an indication to rotate the ultrasound device.
25. The at least one non-transitory computer-readable storage medium of claim 17 , wherein the single mobile device is configured to automatically identify the pose of the ultrasound device at least in part by identifying a location, in the image, of a marker on the ultrasound device.
26. The at least one non-transitory computer-readable storage medium of claim 25 , wherein the single mobile device is configured to automatically identify the pose of the ultrasound device at least in part by identifying a position of the ultrasound device in the image using the identified location of the marker in the image.
27. The at least one non-transitory computer-readable storage medium of claim 17 , wherein the ultrasound device comprises at least one sensor configured to detect movement of the ultrasound device, wherein the single mobile device is configured to automatically identify the pose of the ultrasound device at least in part by using data collected by the at least one sensor.
28. The at least one non-transitory computer-readable storage medium of claim 27 , wherein the at least one sensor comprises a gyroscope.