Mixed reality systems and methods for indicating an extent of a field of view of an imaging device
A mixed reality presentation system determines a device-specific parameter characterizing an extent of a field of view of an active imaging device. The active imaging device captures imagery of an internal view of a body, and the determining of the device-specific parameter is based on data received from the active imaging device. The mixed reality presentation system also determines a spatial pose of the active imaging device as the active imaging device captures the imagery of the internal view of the body. Based on the device-specific parameter and the spatial pose of the active imaging device, the mixed reality presentation system directs a display device to display a shape overlay indicative of the extent of the field of view relative to the body. The shape overlay is displayed together with an external view of the body in a mixed reality presentation. Corresponding systems and methods are also disclosed.
1 . A system comprising:
a mixed reality presentation system comprising:
a memory storing instructions; and
a processor communicatively coupled to the memory and configured to execute the instructions to:
determine, based on data received from an active imaging device configured to capture imagery of an internal view of a patient, a device-specific parameter characterizing an extent of a field of view of the active imaging device,
determine a spatial pose of the active imaging device as the active imaging device captures the imagery of the internal view of the patient, and
based on the device-specific parameter and the spatial pose of the active imaging device, direct a display device to display, together with an external view of the patient captured in real-time by a camera located at a vantage point associated with the external view of the patient while the active imaging device is within the patient and capturing the internal view of the patient, a shape overlay indicative of the extent of the field of view relative to the patient.
2 . The system of claim 1 , wherein the determining of the device-specific parameter comprises:
accessing, based on initialization data received from the active imaging device and representative of a set of device-specific parameters, the device-specific parameter from the set of device-specific parameters represented in the initialization data; or
accessing, based on identification data received from the active imaging device, the device-specific parameter from a stored plurality of device-specific parameters associated with a plurality of different imaging devices including the active imaging device.
3 . The system of claim 1 , wherein the device-specific parameter characterizes the extent of the field of view of the active imaging device by defining an imaging technology employed by the active imaging device.
4 . The system of claim 1 , wherein the device-specific parameter characterizes the extent of the field of view of the active imaging device by defining at least one parameter selected from the group consisting of: a viewing angle of the field of view relative to the active imaging device, a focal length of an image sensor included within the active imaging device, and an aspect ratio of the image sensor.
5 . The system of claim 1 , wherein:
the device-specific parameter characterizes the extent of the field of view of the active imaging device by indicating a viewing angle of the field of view relative to the active imaging device;
the active imaging device is configured to capture the imagery of the internal view from different viewing angles by employing a distal articulation mechanism; and
the directing of the display device to display the shape overlay based on the spatial pose of the active imaging device comprises directing the display device to display the shape overlay based on a current articulation of the distal articulation mechanism.
6 . The system of claim 1 , wherein:
the active imaging device is configured to provide the captured imagery at different zoom levels supported by the active imaging device; and
the directing of the display device to display the shape overlay is further based on a current zoom level, from the different zoom levels supported by the active imaging device, at which the active imaging device is providing the captured imagery.
7 . The system of claim 1 , wherein:
the processor is further configured to execute the instructions to determine, based on the spatial pose and the device-specific parameter, the extent of the field of view relative to the patient; and
the directing of the display device to display the shape overlay is further based on the determining of the extent of the field of view relative to the patient.
8 . The system of claim 1 , further comprising:
a mixed reality media player device configured to be worn on a head of a user and that implements the mixed reality presentation system;
a communication interface included within the mixed reality media player device, the communication interface communicatively coupled to the active imaging device and configured to access the data received from the active imaging device;
a first physical display included within the mixed reality media player device, the first physical display configured to provide a graphical presentation to a first eye of the user when the mixed reality media player device is worn on the head of the user; and
a second physical display included within the mixed reality media player device, the second physical display configured to provide a graphical presentation to a second eye of the user when the mixed reality media player device is worn on the head of the user;
wherein:
the processor and the memory are included within the mixed reality media player device, and
the first and second physical displays included within the mixed reality media player device collectively implement the display device.
9 . The system of claim 1 , wherein:
the processor is further configured to execute the instructions to determine a spatial relationship between the display device and the active imaging device as the active imaging device captures the imagery of the internal view of the patient, wherein the spatial relationship comprises a position relationship or an orientation relationship between the display device and the active imaging device; and
the directing of the display device to display the shape overlay together with the external view of the patient is performed further based on the spatial relationship.
10 . The system of claim 1 , wherein:
the processor is further configured to execute the instructions to determine a depth, relative to the active imaging device, of an anatomical surface depicted in the imagery captured by the active imaging device of the internal view of the patient; and
the shape overlay that is displayed together with the external view of the patient is further indicative of the depth of the anatomical surface.
11 . The system of claim 10 , wherein:
the depth of the anatomical surface is an average depth of a portion of the anatomical surface included within the field of view; or
the depth of the anatomical surface is a depth map representing depth contours of a portion of the anatomical surface included within the field of view.
12 . The system of claim 1 , wherein:
the device-specific parameter characterizing the extent of the field of view indicates that the active imaging device is a stereoscopic imaging device including:
a first image sensor configured to capture the imagery of the patient from a first vantage point, and
a second image sensor configured to capture the imagery of the patient from a second vantage point; and
when only the first image sensor is actively providing the imagery for presentation to a user, the directing of the display device to display the shape overlay together with the external view comprises directing the display device to display a shape overlay corresponding to the first image sensor.
13 . The system of claim 1 , wherein:
the device-specific parameter characterizing the extent of the field of view indicates that the active imaging device is a stereoscopic imaging device including:
a first image sensor configured to capture the imagery of the patient from a first vantage point, and
a second image sensor configured to capture the imagery of the patient from a second vantage point; and
when both the first and second image sensors are actively providing the imagery for presentation to a user, the directing of the display device to display the shape overlay together with the external view comprises directing the display device to display a shape overlay corresponding to both the first and second image sensors.
14 . The system of claim 1 , wherein the processor is further configured to execute the instructions to:
determine, based on the data received from the active imaging device, an additional device-specific parameter characterizing an extent of a potential field of view of the active imaging device; and
based on the additional device-specific parameter, direct the display device to display, together with the external view of the patient, a potential shape overlay indicative of the extent of the potential field of view of the active imaging device relative to the patient.
15 . The system of claim 14 , wherein:
the active imaging device is configured to capture the imagery of the internal view by way of one imaging technology at a time from a plurality of imaging technologies supported by the active imaging device;
the field of view corresponds to a first imaging technology in the plurality of imaging technologies; and
the potential field of view corresponds to a second imaging technology in the plurality of imaging technologies, the second imaging technology distinct from the first imaging technology.
16 . The system of claim 1 , wherein the processor is further configured to execute the instructions to:
determine an additional device-specific parameter characterizing an extent of a potential field of view of a non-active imaging device configured to capture imagery of the internal view of the patient; and
based on the additional device-specific parameter and the spatial pose of the active imaging device, direct the display device to display, together with the external view of the patient and in place of the shape overlay, a potential shape overlay indicative of the extent of the potential field of view of the non-active imaging device relative to the patient.
17 . A method comprising:
determining, by a mixed reality presentation system and based on data received from an active imaging device configured to capture imagery of an internal view of a patient, a device-specific parameter characterizing an extent of a field of view of the active imaging device;
determining, by the mixed reality presentation system, a spatial pose of the active imaging device as the active imaging device captures the imagery of the internal view of the patient; and
directing, by the mixed reality presentation system and based on the device-specific parameter and the spatial pose of the active imaging device, a display device to display, together with an external view of the patient captured in real-time by a camera located at a vantage point associated with the external view of the patient while the active imaging device is within the patient and capturing the internal view of the patient, a shape overlay indicative of the extent of the field of view relative to the patient.
18 . The method of claim 17 , wherein the device-specific parameter characterizes the extent of the field of view of the active imaging device by defining at least one parameter selected from the group consisting of: an imaging technology employed by the active imaging device, a focal length of an image sensor included within the active imaging device, and an aspect ratio of the image sensor.
19 . The method of claim 17 , wherein the device-specific parameter characterizes the extent of the field of view of the active imaging device by indicating a viewing angle of the field of view relative to the active imaging device.
20 . The method of claim 17 , further comprising determining, by the mixed reality presentation system, a depth, relative to the active imaging device, of an anatomical surface depicted in the imagery captured by the active imaging device of the internal view of the patient; and
wherein the shape overlay that is displayed together with the external view of the patient is further indicative of the depth of the anatomical surface.
21 . The method of claim 17 , wherein:
the device-specific parameter characterizing the extent of the field of view indicates that the active imaging device is a stereoscopic imaging device including:
a first image sensor configured to capture the imagery of the patient from a first vantage point, and
a second image sensor configured to capture the imagery of the patient from a second vantage point; and
the directing of the display device to display the shape overlay together with the external view of the patient is further based on which one or both of the first and second image sensors is actively providing the imagery for presentation to a user.
22 . The method of claim 17 , further comprising:
determining, by the mixed reality presentation system an additional device-specific parameter, the additional device-specific parameter characterizing an extent of a potential field of view, the extent of the potential field of view being of the active imaging device or a non-active imaging device; and
directing, by the mixed reality presentation system and based on the additional device-specific parameter, the display device to display, together with the external view of the patient, a potential shape overlay indicative of the extent of the potential field of view relative to the patient.
23 . A non-transitory computer-readable medium storing instructions that, when executed, direct a processor of a mixed reality presentation system to:
determine, based on data received from an active imaging device configured to capture imagery of an internal view of a patient, a device-specific parameter characterizing an extent of a field of view of the active imaging device,
determine a spatial pose of the active imaging device as the active imaging device captures the imagery of the internal view of the patient, and
based on the device-specific parameter and the spatial pose of the active imaging device, direct a display device of the mixed reality presentation system to display, together with an external view of the patient captured in real-time by a camera located at a vantage point associated with the external view of the patient while the active imaging device is within the patient and capturing the internal view of the patient, a shape overlay indicative of the extent of the field of view relative to the patient.