Anatomical feature extraction and presentation using augmented reality
An ultrasound probe captures real-time images of patient anatomy, which are analyzed by a processor to extract salient features pertaining to an anatomical structure. By tracking the location and orientation of the ultrasound probe, a model of that anatomical structure can be created. A visual indication of the position of segments of the anatomical structure can be presented holographically to a user of an augmented reality headset to provide information extracted from the ultrasound imaging, such as holographic display of a model of the anatomical structure at the approximate location of the visual field of the headset corresponding to the physical location of the actual anatomy being viewed by a user, without presenting the entirety of the ultrasound image to the user.
1. A surgical system comprising:
an ultrasound system comprising an ultrasound probe configured to obtain a plurality of two-dimensional ultrasound images of a surgical site;
a head-mounted display;
a tracking system configured to determine a pose of the ultrasound probe; and
a computer system comprising a memory, the computer system configured to:
receive the plurality of two-dimensional ultrasound images from the ultrasound system, wherein the plurality of two-dimensional ultrasound images comprise sequential two-dimensional slices of an anatomical structure at the surgical site obtained as the ultrasound probe was moved relative to the surgical site,
extract a feature from the plurality of two-dimensional ultrasound images, the feature associated with the anatomical structure at the surgical site,
determine the pose of the ultrasound probe via the tracking system for each of the plurality of two-dimensional images,
receive a three-dimensional feature model associated with the anatomical structure from the memory,
update the three-dimensional feature model with the extracted feature based on the determined pose of the ultrasound probe across the plurality of two-dimensional ultrasound images,
determine whether features of the updated three-dimensional feature model are located within a field of view of the head-mounted display, and
display the features of the updated three-dimensional feature model that are located within the field of view of the head-mounted display;
determine whether any of the features of the updated three-dimensional feature model are older than a threshold time period; and
remove the features of the updated three-dimensional feature model that are older than the threshold time period from the field of view of the head-mounted display.
2. The surgical system of claim 1 , wherein the head-mounted display comprises the computer system.
3. The surgical system of claim 1 , wherein the head-mounted display comprises the tracking system.
4. The surgical system of claim 3 , wherein the pose of the ultrasound probe is determined relative to the head-mounted display.
5. The surgical system of claim 1 , wherein:
the pose comprises a first pose;
the tracking system is further configured to determine a second pose of the head-mounted display; and
the computer system is further configured to:
determine the second pose of the head-mounted display via the tracking system,
determine a relative pose of the ultrasound probe relative to the head-mounted display based on the determined first pose of the ultrasound probe and the determined second pose of the head-mounted display, and
determine where features of the updated three-dimensional feature model are located relative to the field of view of the head-mounted display based on the relative pose of the ultrasound probe.
6. The surgical system of claim 1 , wherein the tracking system comprises a camera system configured to identify one or more fiducial markers positioned on the ultrasound probe.
7. The surgical system of claim 1 , wherein the feature comprises a boundary of the anatomical structure.
8. A computer-implemented method of displaying an anatomical structure via a head-mounted display using an ultrasound system comprising an ultrasound probe configured to obtain a plurality of two-dimensional ultrasound images of a surgical site and a tracking system configured to determine a pose of the ultrasound probe, the method comprising:
receiving, by a computer system comprising a memory, the plurality of two-dimensional ultrasound images from the ultrasound system, wherein the plurality of two-dimensional ultrasound images comprise sequential two-dimensional slices of an anatomical structure at the surgical site obtained as the ultrasound probe was moved relative to the surgical site;
extracting, by the computer system, a feature from the plurality of two-dimensional ultrasound images, the feature associated with the anatomical structure at the surgical site;
determining, by the computer system, the pose of the ultrasound probe via the tracking system for each of the plurality of two-dimensional images;
receiving, by the computer system, a three-dimensional feature model associated with the anatomical structure from the memory;
updating, by the computer system, the three-dimensional feature model with the extracted feature based on the determined pose of the ultrasound probe across the plurality of two-dimensional ultrasound images;
determining, by the computer system, whether features of the updated three-dimensional feature model are located within a field of view of the head-mounted display; and
displaying, by the computer system the features of the updated three-dimensional feature model that are located within the field of view of the head-mounted display;
determining, by the computer system, whether any of the features of the updated three-dimensional feature model are older than a threshold time period; and
removing, by the computer system, the features of the updated three-dimensional feature model that are older than the threshold time period from the field of view of the head-mounted display.
9. The computer-implemented method of claim 8 , wherein the head-mounted display comprises the computer system.
10. The computer-implemented method of claim 8 , wherein the head-mounted display comprises the tracking system.
11. The computer-implemented method of claim 10 , wherein the pose of the ultrasound probe is determined relative to the head-mounted display.
12. The computer-implemented method of claim 8 , wherein:
the pose comprises a first pose;
the tracking system is further configured to determine a second pose of the head-mounted display; and
the method further comprises:
determining, by the computer system, the second pose of the head-mounted display via the tracking system,
determining, by the computer system, a relative pose of the ultrasound probe relative to the head-mounted display based on the determined first pose of the ultrasound probe and the determined second pose of the head-mounted display, and
determining, by the computer system, where features of the updated three-dimensional feature model are located relative to the field of view of the head-mounted display based on the relative pose of the ultrasound probe.
13. The computer-implemented method of claim 8 , wherein the tracking system comprises a camera system configured to identify one or more fiducial markers positioned on the ultrasound probe.