Granted Patent B2

US 12,295,784 · App. 17/799,826 · Granted May 13, 2025

Inventors: Jonathan Silva (St. Louis, MO); Christopher Andrews (Chesterfield, MO); Jennifer Silva (St. Louis, MO); Zahid Iqbal (Kirkwood, MO)

Assignee: Washington University

A61B8/42A61B34/20A61B8/4416

Abstract

Claims (66)

1. A mixed reality (MR) visualization system, comprising:

an MR device comprising a holographic display configured to display a holographic image to an operator;

a hand-held ultrasound imaging device configured to obtain a real-time ultrasound image of a subject's anatomy; and

a computing device communicatively coupled to the MR device and the hand-held ultrasound imaging device, the computing device comprising a non-volatile memory and a processor, wherein the computing device is configured to:

receive the real-time ultrasound image;

determine a real-time 3D position and orientation of the hand-held ultrasound imaging device;

generate a modified real-time ultrasound image by modifying the real-time ultrasound image to correspond to the real-time 3D position and orientation of the hand-held ultrasound imaging device; and

transmit the modified real-time ultrasound image to the MR device for display as the holographic image positioned at a predetermined location relative to the hand-held ultrasound imaging device.

2. The system of claim 1 , wherein:

the ultrasound imaging device comprises a tracking device for tracking the real-time 3D position and orientation of the hand-held ultrasound imaging device; and

the computing device is configured to determine real-time 3D position and orientation of the hand-held ultrasound imaging device based at least in part on the tracking device.

3. The system of claim 2 , wherein:

the tracking device comprises an image registration target attached to the hand-held ultrasound imaging device;

the MR device further comprises a front-facing camera configured to obtain a real-time image of the image registration target; and

the computing device is further configured to determine the real-time 3D position and orientation of the hand-held ultrasound imaging device based at least in part on the real-time image of the image registration target.

4. The system of claim 2 , wherein the tracking device includes an inertial measurement unit (IMU) configured to track the real-time 3D position and orientation of the hand-held ultrasound imaging device, and the computing device is further configured to determine the real-time 3D position and orientation of the hand-held ultrasound imaging device based at least in part on the real-time 3D position and orientation of the hand-held ultrasound imaging device tracked by the IMU.

5. The system of claim 1 , further comprising an external tracking system configured to track the real-time 3D position and orientation of the hand-held ultrasound imaging device, the external tracking system selected from the group consisting of an IR camera array, a depth camera, an electromagnetic tracking array, and any combination thereof, wherein the hand-held ultrasound imaging device further comprises at least one marker affixed to the hand-held ultrasound imaging device, the at least one marker detectable by the external tracking system, and the computing device is configured to determine real-time 3D position and orientation of the hand-held ultrasound imaging device based at least in part on the real-time 3D position and orientation of the hand-held ultrasound imaging device detected by the external tracking system.

6. The system of claim 5 , wherein the tracking device comprises one or both of an image registration target and an inertial measurement unit, and the computing device is configured to calculate a composite real-time 3D position and orientation of the hand-held ultrasound imaging device using a Kalman filter to combine at least two of:

the real-time 3D position and orientation of the hand-held ultrasound imaging device based on a real-time image of the image registration target captured by a camera of the MR device;

the real-time 3D position and orientation of the hand-held ultrasound imaging device tracked by the IMU; and

the real-time 3D position and orientation of the hand-held ultrasound imaging device tracked by the external tracking system.

7. The system of claim 1 , wherein the computing device is further configured to:

receive a real-time 3D position of a surgical instrument;

register the real-time 3D position of the surgical instrument relative to the real-time ultrasound image device; and

generate the modified real-time ultrasound image by further modifying the real-time ultrasound image to include an icon superimposed on the real-time ultrasound image, the icon indicative of the real-time 3D position of the surgical instrument.

8. The system of claim 7 , wherein the real-time 3D position of the surgical instrument is received from an instrument position sensor of a surgical instrument system operatively coupled to the computing device.

9. The system of claim 1 , wherein the MR device is further configured to detect a cue produced by an operator.

10. The system of claim 9 , wherein the cue is selected from one or more of a gesture, an eye movement, a voice comment, a facial expression, and a head movement.

11. The system of claim 9 , wherein the computing device is further configured to modify the holographic image in response to the cue produced by the operator, the modification comprising at least one of a zoom, a rotation, a translation, a generation of a cross section, and a change of rendering of the real-time ultrasound image.

12. The system of claim 9 , wherein the computing device is further configured to modify the holographic image in response to the cue produced by the operator, the modification comprising at least one of:

simultaneously displaying two or more user-selected ultrasound images using the holographic display;

creating at least one icon indicative of a user-selected position or area on at least one real-time ultrasound image for display on the holographic display; and

combining each of the at least one icons to produce a line or volume for display on the holographic display.

13. A computer-implemented method of MR visualization of a real-time ultrasound image using a system including a computing device communicatively coupled to a hand-held ultrasound imaging device and a mixed reality (MR) device including a holographic display, the hand-held ultrasound imaging device configured to obtain the real-time ultrasound image, the method comprising:

receiving, using a computing device, the real-time ultrasound image of a subject's anatomy;

determining, using the computing device, the real-time 3D position and orientation of the hand-held ultrasound imaging device;

generating, using the computing device, a modified real-time ultrasound image by modifying the real-time ultrasound image to correspond to the real-time 3D position and orientation of the hand-held ultrasound imaging device; and

transmitting, using the computing device, the modified real-time ultrasound image to the MR device for display as a holographic image positioned at a predetermined location relative to the hand-held ultrasound imaging device.

14. The method of claim 13 , wherein:

the ultrasound imaging device comprises a tracking device for tracking the real-time 3D position and orientation of the hand-held ultrasound imaging device; and

determining the real-time 3D position and orientation of the hand-held ultrasound imaging device comprises determining the real-time 3D position and orientation of the hand-held ultrasound imaging device based at least in part on the tracking device.

15. The method of claim 14 , wherein:

the tracking device comprises an image registration target attached to the hand-held ultrasound imaging device, the MR device further comprises a front-facing camera configured to obtain a real-time image of the image registration target; and

determining the real-time 3D position and orientation of the hand-held ultrasound imaging device comprises determining the real-time 3D position and orientation of the hand-held ultrasound imaging device based at least in part on the real-time image of the image registration target.

16. The method of claim 14 , wherein:

the tracking device includes an inertial measurement unit (IMU) configured to track the real-time 3D position and orientation of the hand-held ultrasound imaging device; and

determining the real-time 3D position and orientation of the hand-held ultrasound imaging device comprises determining the real-time 3D position and orientation of the hand-held ultrasound imaging device based at least in part on the real-time 3D position and orientation of the hand-held ultrasound imaging device tracked by the IMU.

17. The method of claim 13 , wherein:

the system further comprises an external tracking system configured to track the real-time 3D position and orientation of the hand-held ultrasound imaging device, the external tracking system selected from the group consisting of an IR camera array, a depth camera, an electromagnetic tracking array, and any combination thereof, wherein the hand-held ultrasound imaging device further comprises at least one marker affixed to the hand-held ultrasound imaging device, the at least one marker detectable by the external tracking system; and

determining the real-time 3D position and orientation of the hand-held ultrasound imaging device comprises determining the real-time 3D position and orientation of the hand-held ultrasound imaging device based at least in part on the real-time 3D position and orientation of the hand-held ultrasound imaging device detected by the external tracking system.

18. The method of claim 17 , wherein the tracking device comprises one or both of an image registration target and an inertial measurement unit, and determining the real-time 3D position and orientation of the hand-held ultrasound imaging device comprises determining the real-time 3D position and orientation of the hand-held ultrasound imaging device by determining a composite real-time 3D position and orientation of the hand-held ultrasound imaging device using a Kalman filter to combine at least two of:

the real-time 3D position and orientation of the hand-held ultrasound imaging device based on a real-time image of the image registration target captured by a camera of the MR device;

the real-time 3D position and orientation of the hand-held ultrasound imaging device tracked by the IMU; and

the real-time 3D position and orientation of the hand-held ultrasound imaging device tracked by the external tracking system.

19. The method of claim 13 , further comprising:

receiving, using the computing device, a real-time 3D position of a surgical instrument;

registering, using the computing device, the real-time 3D position of the surgical instrument relative to the real-time ultrasound image device; and

generating, using the computing device, the modified real-time ultrasound image by further modifying the real-time ultrasound image to include an icon superimposed on the real-time ultrasound image, the icon indicative of the real-time 3D position of the surgical instrument.

20. The method of claim 19 , wherein receiving the real-time 3D position of a surgical instrument comprises receiving the real-time 3D position of the surgical instrument from an instrument position sensor of a surgical instrument system operatively coupled to the computing device.

21. The method of claim 13 , further comprising receiving, using the computing device, a cue produced by an operator and detected by the MR device.

22. The method of claim 21 , wherein receiving the cue comprises receiving a cue selected from one or more of a gesture, an eye movement, a voice comment, a facial expression, and a head movement.

23. The method of claim 21 , further comprising modifying, using the computing device, the holographic image in response to the cue produced by the operator, the modification comprising at least one of a zoom, a rotation, a translation, a generation of a cross section, and a change of rendering of the real-time ultrasound image.

24. The method of claim 21 , further comprising modifying, using the computing device, the holographic image in response to the cue produced by the operator, the modification comprising at least one of:

simultaneously displaying two or more user-selected ultrasound images using the holographic display;

creating at least one icon indicative of a user-selected position or area on at least one real-time ultrasound image for display on the holographic display; and

combining each of the at least one icons to produce a line or volume for display on the holographic display.

Assignments (2)

Continuity (2)

References Cited (13)