Augmenting real-time views of a patient with three-dimensional data
Augmenting real-time views of a patient with three-dimensional (3D) data. In one embodiment, a method may include identifying 3D data for a patient with the 3D data including an outer layer and multiple inner layers, determining virtual morphometric measurements of the outer layer from the 3D data, registering a real-time position of the outer layer of the patient in a 3D space, determining real-time morphometric measurements of the outer layer of the patient, automatically registering the position of the outer layer from the 3D data to align with the registered real-time position of the outer layer of the patient in the 3D space using the virtual morphometric measurements and using the real-time morphometric measurements, and displaying, in an augmented reality (AR) headset, one of the inner layers from the 3D data projected onto real-time views of the outer layer of the patient.
1. A method for augmenting real-time, non-image actual views of a patient with three-dimensional (3D) data, the method comprising:
identifying 3D data for a patient, the 3D data including an outer layer of the patient and multiple inner layers of the patient; and
displaying, in an augmented reality (AR) headset, one of the inner layers of the patient from the 3D data projected onto real-time, non-image actual views of the outer layer of the patient, the projected inner layer of the patient from the 3D data being confined within a volume of a virtual spatial difference 3D shape.
2. The method as recited in claim 1 , further comprising:
capturing, in real-time, real-time 3D video data of the patient, the real-time 3D video data including the outer layer of the patient and the multiple inner layers of the patient; and
displaying, in the AR headset and in real-time, one of the inner layers of the patient from the real-time 3D video data projected onto the real-time, non-image actual views of the outer layer of the patient.
3. The method as recited in claim 2 , wherein the projected inner layer of the real-time 3D video data depicts an internal organ of the patient in motion within the outer layer of the patient.
4. One or more non-transitory computer-readable media storing one or more programs that are configured, when executed, to cause one or more processors to perform the method as recited in claim 2 .
5. The method as recited in claim 1 , further comprising:
positioning the patient in a different position than the patient was in when the 3D data of the patient was captured;
deforming the 3D data to match the different position of the patient; and
displaying, in the AR headset, one of the inner layers of the patient from the deformed 3D data projected onto the real-time, non-image actual views of the outer layer of the patient.
6. The method as recited in claim 5 , wherein:
the positioning includes positioning the patient in a different orientation than the patient was in when the 3D data of the patient was captured; and
the deforming includes deforming the 3D data to match the different orientation of the patient.
7. The method as recited in claim 1 , wherein:
the virtual spatial difference 3D shape is a virtual spatial difference 3D box; and
the virtual spatial difference 3D box includes a top side, a bottom side, a left side, a right side, a front side, and a back side.
8. The method as recited in claim 1 , further comprising:
automatically tracking a real-time position of an object with respect to a real-time position of the outer layer of the patient; and
while a portion of the object is inserted into the patient through the outer layer of the patient, displaying, in the AR headset, a virtual portion of the object projected into the projected inner layer of the patient.
9. The method as recited in claim 8 , wherein:
the object is formed from a material that a sensor of the AR headset is able to sense even after the object has been inserted into the patient through the outer layer of the patient; and
the automatically tracking includes the sensor of the AR headset automatically tracking the real-time position of the object with respect to the real-time position of the outer layer of the patient.
10. The method as recited in claim 1 , further comprising:
generating, in the AR headset, a virtual user interface that includes options for altering the display of the projected inner layer of the patient from the 3D data;
displaying, in the AR headset, the virtual user interface projected onto real-time, non-image actual views; and
updating, in real-time, the displaying of the virtual user interface, in the AR headset, to cause the virtual user interface to be continually oriented perpendicularly to a focal orientation of the AR headset.
11. The method as recited in claim 10 , wherein:
the perpendicular orientation of the virtual user interface to the focal orientation of the AR headset causes the virtual user interface to constantly face a user wearing the AR headset head-on regardless of a current focal orientation of the AR headset, even as the user faces toward the patient or faces away from the patient.
12. The method of claim 1 , wherein:
the virtual spatial difference 3D shape is configured to be controlled to toggle between displaying and hiding lines of the virtual spatial difference 3D shape; and
the virtual spatial difference 3D shape is configured to be controlled to reposition two-dimensional (2D) slices and/or 3D slices of the projected inner layer of the patient from the 3D data.
13. The method of claim 1 , wherein lines of the virtual spatial difference 3D shape are displayed.
14. The method of claim 1 , wherein lines of the virtual spatial difference 3D shape are hidden.
15. One or more non-transitory computer-readable media storing one or more programs that are configured, when executed, to cause one or more processors to perform the method as recited in claim 5 .
16. One or more non-transitory computer-readable media storing one or more programs that are configured, when executed, to cause one or more processors to perform the method as recited in claim 1 .
17. A method for augmenting real-time, non-image actual views of a patient with three-dimensional (3D) data, the method comprising:
identifying 3D data for a patient, the 3D data including an outer layer of the patient and multiple inner layers of the patient;
automatically registering a virtual position of the outer layer of the patient from the 3D data to align with a real-time position of the outer layer of the patient;
displaying, in an augmented reality (AR) headset, one of the inner layers of the patient from the 3D data projected onto real-time, non-image actual views of the outer layer of the patient;
generating a numeric confidence score that the automatic registering is correct; and
displaying, in the AR headset, the numeric confidence score to a user.
18. The method as recited in claim 17 , wherein the numeric confidence score is a number between 1 and 100 that indicates a level of confidence that the outer layer of the 3D data matches the outer layer of the patient.
19. One or more non-transitory computer-readable media storing one or more programs that are configured, when executed, to cause one or more processors to perform the method as recited in claim 17 .
20. One or more non-transitory computer-readable media storing one or more programs that are configured, when executed, to cause one or more processors to perform the method as recited in claim 8 .
21. A method for augmenting real-time, non-image actual views of a patient with three-dimensional (3D) data, the method comprising:
identifying 3D data for a patient, the 3D data including an outer layer of the patient and multiple inner layers of the patient;
displaying, in an augmented reality (AR) headset, one of the inner layers of the patient from the 3D data projected onto real-time, non-image actual views of the outer layer of the patient;
generating, in the AR headset, a virtual cursor and/or a virtual user interface that includes options for altering the display of the projected inner layer of the patient from the 3D data;
displaying, in the AR headset, the virtual cursor and/or the virtual user interface projected onto real-time, non-image actual views when it is determined that a focal orientation of the AR headset is focused elsewhere than on the patient; and
hiding, in the AR headset, the virtual cursor and/or the virtual user interface when it is determined that the focal orientation of the AR headset is focused on the patient.
22. The method as recited in claim 21 , wherein:
it is determined that the focal orientation of the AR headset is focused elsewhere than on the patient when the AR headset is above horizontal; and
it is determined that the focal orientation of the AR headset is focused on the patient when the AR headset is below horizontal.
23. One or more non-transitory computer-readable media storing one or more programs that are configured, when executed, to cause one or more processors to perform the method as recited in claim 21 .
24. A method for augmenting real-time, non-image actual views of a patient with three-dimensional (3D) data, the method comprising:
identifying 3D data for a patient, the 3D data including an outer layer of the patient and multiple inner layers of the patient;
displaying, in an augmented reality (AR) headset, one of the inner layers of the patient from the 3D data projected onto real-time, non-image actual views of the outer layer of the patient;
generating, in the AR headset, a virtual user interface that includes options for altering the display of the projected inner layer of the patient from the 3D data;
displaying, in the AR headset, the virtual user interface projected onto real-time, non-image actual views;
determining, in real-time, a distance of the patient from the AR headset; and
updating, in real-time, the displaying of the virtual user interface, in the AR headset, to cause the virtual user interface to be continually positioned at a focal distance from the AR headset that is about equal to the real-time distance of the patient from the AR headset.
25. The method as recited in claim 24 , wherein the real-time distance of the patient from the AR headset is the real-time distance to: the center of the patient, to the center of an area of focus of the patient, or to a slice of the 3D data currently being viewed on the patient.
26. One or more non-transitory computer-readable media storing one or more programs that are configured, when executed, to cause one or more processors to perform the method as recited in claim 24 .
27. One or more non-transitory computer-readable media storing one or more programs that are configured, when executed, to cause one or more processors to perform the method as recited in claim 10 .