Instrument-based registration and alignment for augmented reality environments
Various embodiments of an apparatus, methods, systems and computer program products described herein are directed to Registration Engine. The Registration Engine tracks positions of a fiducial marker relative to positions of an Augmented Reality (AR) headset device worn by a user. The Registration Engine receives respective registration landmarks that correspond with locations of a physical region of a patient's anatomy and the tracked positions of the fiducial marker. The Registration Engine generates an AR display of medical data at the AR headset device in alignment with the physical region of a patient's anatomy that corresponds with the respective registration landmarks.
1. A computer-implemented method, comprising:
tracking one or more positions and orientations of a fiducial marker relative to one or more positions and orientations of an Augmented Reality (AR) headset device worn by a user;
prior to generation of an AR display of medical data at the AR headset device:
receiving one or more physical landmark registration notifications based on particular coordinates of a tip of an instrument selected by the user, the particular coordinates (i) being relative to a predefined fixed reference point in three-dimensional (3D) space external to the AR headset device and (ii) represent respective physical landmarks at a region of the patient's anatomy, the fiducial marker disposed on the instrument;
receiving one or more virtual landmark registration notifications based on selection by the user of one or more particular display locations on a display of medical data, the display of medical data based on the region of the patient's anatomy at which the respective physical landmarks are located; and
generating an AR display of medical data at the AR headset device in alignment with the physical region of the patient's anatomy that corresponds with the respective physical and virtual landmark registrations.
2. The computer-implemented method of claim 1 , wherein generating an AR display of medical data at the AR headset device in alignment with the physical region of the patient's anatomy comprises:
generating a first AR display based on medical data related to the physical region of the patient's anatomy; and
displaying the first AR display at the AR headset device to present in the user's field of vision a stationary visualization of the medical data overlayed upon the physical region of the patient's anatomy.
3. The computer-implemented method of claim 2 , wherein generating a first AR display base on medical data related to the physical region of the patient's anatomy comprises:
determining a mapping from a predefined fixed reference point in three-dimensional (3D) space external to the AR headset device to the respective registration landmarks selected by the user at the physical region of a patient's anatomy; and
applying the mapping to a model of the medical data related to the physical region of the patient's anatomy to identify one or more portions of the model of the medical data to be rendered as the stationary visualization of the medical data overlayed upon the physical region of the patient's anatomy.
4. The computer-implemented method of claim 3 , wherein tracking one or more poses of a fiducial marker relative to one or more positions of an Augmented Reality (AR) headset device worn by a user comprises:
capturing one or more images via one or more cameras disposed on the AR headset device as the headset device moves;
calculating a first spatial transformation based on the one or more captured images and a predefined fixed reference point in 3D space external to the AR headset device via one or more simultaneous localization and mapping (SLAM) algorithms executed at the AR headset device; and
generating device pose data resulting from the spatial transformation, the device pose data representing a physical orientation and a position of the AR headset device.
5. The computer-implemented method of claim 4 , further comprising:
capturing one or more images of the fiducial marker via the one or more cameras disposed on the AR headset device as the headset device moves and as the fiducial marker moves;
calculating a second spatial transformation based on the one or more captured images of the fiducial marker and camera data via one or more fiducial tracking algorithms executed at the AR headset device; and
generating respective coordinates representing one or more physical orientations and positions of the fiducial marker in the 3D space.
6. The computer-implemented method of claim 5 , further comprising:
calculating a third spatial transformation based on a fixed distance of an instrument between the fiducial marker and a tip of the instrument to generate respective coordinates representing one or more physical orientations and positions of tip of the instrument in the 3D space.
7. The computer-implemented method of claim 1 , wherein generating an AR display of medical data at the AR headset device in alignment with the physical region of the patient's anatomy that corresponds with the respective physical and virtual landmark registrations comprises:
at one or more computer systems located remotely from the AR headset device:
receiving one or more portions of AR headset device pose data representing a current physical orientation of the AR headset device with respect to a current placement of the physical region of the patient's anatomy, the AR headset device pose data and the current placement of the physical region of the patient's anatomy defined relative to the predefined fixed reference point in three-dimensional (3D) space;
identifying a current view of one or more portions of the physical region of the patient's anatomy from a current perspective of the AR headset device in accordance with the AR headset device pose data and the current placement of the physical region of the patient's anatomy;
identifying one or more portions of the medical data, bounded by the virtual landmarks registrations, that correspond to the current view of the one or more portions of the physical region;
generating one or more AR display base images based on a rendering of identified one or more portions of the medical data; and
sending the one or more AR display base images to the AR headset device.
8. A system comprising one or more processors, and a non-transitory computer-readable medium including one or more sequences of instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
tracking one or more positions and orientations of a fiducial marker relative to one or more positions and orientations of an Augmented Reality (AR) headset device worn by a user;
prior to generation of an AR display of medical data at the AR headset device:
receiving one or more physical landmark registration notifications based on particular coordinates of a tip of an instrument selected by the user, the particular coordinates (i) being relative to a predefined fixed reference point in three-dimensional (3D) space external to the AR headset device and (ii) represent respective physical landmarks at a region of the patient's anatomy, the fiducial marker disposed on the instrument;
receiving one or more virtual landmark registration notifications based on selection by the user of one or more particular display locations on a display of medical data, the display of medical data based on the region of the patient's anatomy at which the respective physical landmarks are located; and
generating an AR display of medical data at the AR headset device in alignment with the physical region of the patient's anatomy that corresponds with the respective physical and virtual landmark registrations.
9. The system of claim 8 , wherein generating an AR display of medical data at the AR headset device in alignment with the physical region of the patient's comprises:
generating a first AR display based on medical data related to the physical region of the patient's anatomy; and
displaying the first AR display at the AR headset device to present in the user's field of vision a stationary visualization of the medical data overlayed upon the physical region of the patient's anatomy.
10. The system of claim 9 , wherein generating a first AR display base on medical data related to the physical region of the patient's anatomy comprises:
determining a mapping from a predefined fixed reference point in three-dimensional (3D) space external to the AR headset device to the respective registration landmarks selected by the user at the physical region of a patient's anatomy; and
applying the mapping to a model of the medical data related to the physical region of the patient's anatomy to identify one or more portions of the model of the medical data to be rendered as the stationary visualization of the medical data overlayed upon the physical region of the patient's anatomy.
11. The system of claim 10 , wherein tracking one or more positions of a fiducial marker relative to one or more positions of an Augmented Reality (AR) headset device worn by a user comprises:
capturing one or more images via one or more cameras disposed on the AR headset device as the headset device moves;
calculating a first spatial transformation based on the one or more captured images and a predefined fixed reference point in 3D space external to the AR headset device via one or more simultaneous localization and mapping (SLAM) algorithms executed at the AR headset device; and
generating device pose data resulting from the spatial transformation, the device pose data representing a physical orientation and a position of the AR headset device.
12. The system of claim 11 , further comprising:
capturing one or more images of the fiducial marker via the one or more cameras disposed on the AR headset device as the headset device moves and as the fiducial marker moves;
calculating a second spatial transformation based on the one or more captured images of the fiducial marker and camera data via one or more fiducial tracking algorithms executed at the AR headset device; and
generating respective coordinates representing one or more physical orientations and positions of the fiducial marker in the 3D space.
13. The system of claim 12 , further comprising:
calculating a third spatial transformation based on a fixed distance of an instrument between the fiducial marker and a tip of the instrument to generate respective coordinates representing one or more physical orientations and positions of tip of the instrument in the 3D space.
14. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions for:
tracking one or more positions and orientations of a fiducial marker relative to one or more positions and orientations of an Augmented Reality (AR) headset device worn by a user;
prior to generation of an AR display of medical data at the AR headset device:
receiving one or more physical landmark registration notifications based on particular coordinates of a tip of an instrument selected by the user, the particular coordinates (i) being relative to a predefined fixed reference point in three-dimensional (3D) space external to the AR headset device and (ii) represent respective physical landmarks at a region of the patient's anatomy, the fiducial marker disposed on the instrument;
receiving one or more virtual landmark registration notifications based on selection by the user of one or more particular display locations on a display of medical data, the display of medical data based on the region of the patient's anatomy at which the respective physical landmarks are located; and
generating an AR display of medical data at the AR headset device in alignment with the physical region of the patient's anatomy that corresponds with the respective physical and virtual landmark registrations.
15. The computer program product of claim 14 , wherein generating an AR display of medical data at the AR headset device in alignment with the physical region of the patient's anatomy comprises:
generating a first AR display based on medical data related to the physical region of the patient's anatomy; and
displaying the first AR display at the AR headset device to present in the user's field of vision a stationary visualization of the medical data overlayed upon the physical region of the patient's anatomy.
16. The computer program product of claim 15 , wherein generating a first AR display base on medical data related to the physical region of the patient's anatomy comprises:
determining a mapping from a predefined fixed reference point in three-dimensional (3D) space external to the AR headset device to the respective registration landmarks selected by the user at the physical region of a patient's anatomy; and
applying the mapping to a model of the medical data related to the physical region of the patient's anatomy to identify one or more portions of the model of the medical data to be rendered as the stationary visualization of the medical data overlayed upon the physical region of the patient's anatomy.
17. The computer program product of claim 16 , wherein tracking one or more positions of a fiducial marker relative to one or more positions of an Augmented Reality (AR) headset device worn by a user comprises:
capturing one or more images via one or more cameras disposed on the AR headset device as the headset device moves;
calculating a first spatial transformation based on the one or more captured images and a predefined fixed reference point in 3D space external to the AR headset device via one or more simultaneous localization and mapping (SLAM) algorithms executed at the AR headset device; and
generating device pose data resulting from the spatial transformation, the device pose data representing a physical orientation and a position of the AR headset device.
18. The computer program product of claim 17 , further comprising:
capturing one or more images of the fiducial marker via the one or more cameras disposed on the AR headset device as the headset device moves and as the fiducial marker moves;
calculating a second spatial transformation based on the one or more captured images of the fiducial marker and camera data via one or more fiducial tracking algorithms executed at the AR headset device; and
generating respective coordinates representing one or more physical orientations and positions of the fiducial marker in the 3D space.
19. The computer program product of claim 18 , further comprising:
calculating a third spatial transformation based on a fixed distance of an instrument between the fiducial marker and a tip of the instrument to generate respective coordinates representing one or more physical orientations and positions of tip of the instrument in the 3D space.