Spatially-aware displays for computer-assisted interventions
Described herein are systems, methods, and techniques for spatially-aware displays for computer-assisted interventions. A Fixed View Frustum technique renders computer images on the display using a perspective based on a virtual camera having a field-of-view facing the display and automatically updates the virtual position of the virtual camera in response to adjusting the pose of the display. A Dynamic Mirror View Frustum technique renders computer images on the display using a perspective based on a field-of-view of a virtual camera that has a virtual position behind the display device. The virtual position of the virtual camera is dynamically updated in response to movement of a user's viewpoint located in front of the display device. Slice visualization techniques are also described herein for use with the Fixed View Frustum and Dynamic Mirror View Frustum techniques.
1. A system for aiding in interaction with a physical object, the system comprising:
a display device defining a plane and being located on a first side of the physical object; and
a navigation system coupled to a control system and being configured to:
register computer images to the physical object;
track a pose of the physical object and the display device in a common coordinate system; and
control the display device to render the registered computer images according to the tracked pose of the physical object and the display device; and
wherein a perspective of the rendering is based on a virtual camera that has a virtual position located on a second side of the physical object opposite to the first side, and wherein the virtual camera has a field-of-view that faces the plane of display device, and wherein the virtual position of the virtual camera updates automatically in response to adjustment to the pose of the display device; and
wherein the virtual position of the virtual camera is located at a predetermined distance from the plane of display device and wherein the predetermined distance updates automatically.
2. The system of claim 1 , wherein the computer images of the physical object are derived from a 3D model, and the control system is configured to control the display device to display one or more slices of the 3D model depending upon the tracked pose of the physical object and the display device.
3. The system of claim 2 , wherein the one or more slices are sliced at planes parallel to the plane of the display device.
4. The system of claim 2 , wherein the control system is configured to control the display device to automatically change the one or more slices to other slices in response to one or more of the following: the tracked pose of the display device; the tracked pose of the physical object; a tracked pose of a surgical instrument; and a tracked pose of a user's viewpoint.
5. The system of claim 1 , wherein the predetermined distance updates automatically in response to at least one or more of the following: the tracked pose of the display device, the tracked pose of the physical object, a tracked pose of a surgical tool; a tracked pose of a user's viewpoint; a type of surgical procedure; and a certain step of a surgical procedure.
6. The system of claim 1 , wherein an X-Y placement of the virtual position of the virtual camera relative to the plane of the display device updates automatically in response to at least one or more of the following: the tracked pose of the display device, the tracked pose of the physical object, a tracked pose of a surgical tool; a tracked pose of a user's viewpoint; a type of surgical procedure; and a certain step of a surgical procedure.
7. The system of claim 1 , wherein an X-Y placement of the virtual position of the virtual camera relative to the plane of the display device is fixed.
8. The system of claim 1 , wherein the pose of the display device is manually adjustable, and the virtual position of the virtual camera updates automatically in response to manual adjustment to the pose of the display device.
9. The system of claim 1 , comprising one or more actuators coupled to the display device and the control system being configured to control the one or more actuators to adjust the pose of the display device.
10. The system of claim 9 , comprising an input device coupled to the control system and the control system is configured to receive a command from the input device and control the one or more actuators to adjust the pose of the display device in accordance with the command.
11. The system of claim 9 , comprising a surgical instrument including one or more trackable features, and wherein the navigation system is configured to track a pose of the surgical instrument in the common coordinate system and control the display device to display an image of the surgical instrument, and wherein the control system is configured to control the one or more actuators to adjust the pose of the display device based on the tracked pose of the surgical instrument.
12. The system of claim 9 , comprising a viewpoint tracking system coupled to the navigation system and being configured to track a pose of a user's viewpoint in the common coordinate system and wherein the control system is configured to control the one or more actuators to adjust the pose of the display device based on the tracked pose related to the user's viewpoint.
13. A method of operating a system for aiding in interaction with a physical object, the system comprising a display device defining a plane and being located on a first side of the physical object, and a navigation system coupled to a control system, the method comprising:
registering computer images to the physical object;
tracking a pose of the physical object and the display device in a common coordinate system; and
controlling the display device to render the registered computer images according to the tracked pose of the physical object and the display device; and
wherein a perspective of the rendering is based on a virtual camera having a virtual position located on a second side of the physical object opposite to the first side, and wherein the virtual camera has a field-of-view facing the plane of display device, and automatically updating the virtual position of the virtual camera in response to adjusting the pose of the display device; and
wherein the virtual position of the virtual camera is located at a predetermined distance from the plane of display device and wherein the predetermined distance updates automatically.