Mixed reality (MR)-based optical device
Systems and methods for generating a Mixed Reality (MR)-based optical device (e.g., MR-based slit lamp and/or MR-based microscope) are provided. The system utilizes two red, green, and blue (RGB) cameras of the MR device, substituting for operator's eyes to observe through the eyepieces of a slit lamp biomicroscope or a conventional optical microscope. The operator, wearing an MR headset, observes a three-dimensional (3D) visualization of examined eye region or, in the case of a microscope, the sample. The system enables slit lamp examinations to be conducted in a manner suitable for telemedicine, allowing multiple users to simultaneously view real-time images from the slit lamp and/or a microscope.
1 . A system for generating a Mixed Reality (MR)-based optical device, the system comprising:
a) a plurality of hardware modules integrated with a first optical device and configured to capture real-time stereoscopic data through an optical path, the real-time stereoscopic data comprising both image data and video data, captured via the first optical device;
b) a plurality of MR display devices operatively coupled to the plurality of hardware modules and configured to render the real-time stereoscopic data in an immersive environment; and
c) a plurality of software modules configured to acquire, process, and transmit the real-time stereoscopic data to the plurality of MR display devices,
the first optical device comprising at least one of a slit lamp and a microscope,
the MR-based optical device comprising at least one of an MR-based slit lamp and an MR-based microscope,
the plurality of software modules comprising:
i) a local application configured to support clinically intuitive interaction between a local operator and a patient, concurrently producing an enhanced, geometrically accurate three-dimensional (3D) visualization of an ocular region under examination;
ii) a streaming application configured to allow transmission of the real-time stereoscopic data to remote physicians equipped with MR, augmented reality (AR), or virtual reality (VR) headsets, thereby enabling collaborative clinical examinations across distributed locations; and
iii) a remote application configured to interface with one or more of the plurality of MR display devices, thereby enabling reliable data acquisition and interactive functionality, and
the plurality of software modules further configured to record and store the real-time stereoscopic data for on-demand review, thereby enhancing telemedicine capabilities in ophthalmology and improving emergency response in ophthalmic clinics.
2 . The system according to claim 1 , the plurality of hardware modules comprising:
mounting hardware structurally connected to the first optical device configured to support integration of cameras across multiple models of the first optical device; and
communication hardware configured to enable bidirectional data exchange with the plurality of MR display devices, thereby establishing a flexible platform interoperable with existing clinical optical device systems,
the clinical optical device systems comprising at least one of a clinical slit lamp system and a clinical microscope system.
3 . The system according to claim 2 , the mounting hardware comprising:
a structure configured to facilitate efficient attachment to and detachment from the first optical device;
a locking component configured to secure and maintain mechanical stability; and
fine adjustment assemblies configured to align the cameras with respect to image centering, focal distance, and rotational orientation,
the mounting hardware configured to:
be compatible with both portable handheld and table-mounted models;
achieve optimal alignment between optical axes of eyepieces of the first optical device and the cameras; and
eliminate artifacts including vignetting, distortion, or partial image loss.
4 . The system according to claim 2 , the cameras comprising a dual-channel video camera setup including two red, green, and blue (RGB) cameras configured to capture and stream the real-time stereoscopic data synchronously for immersive viewing,
each camera being mounted onto the first optical device, and
each camera being configured to preserve image resolution and depth perception consistent with the inherent optical characteristics of the first optical device.
5 . The system according to claim 2 , the communication hardware comprising:
integrated microphones and speakers within the plurality of MR display devices, configured to facilitate direct audio communication without the use of external devices; and
communication components configured to optimize real-time data exchange by maintaining a streaming latency of less than 0.1 seconds between the plurality of MR display devices, a local computing device, and a remote streaming server.
6 . The system according to claim 1 , the local application comprising:
a stream client configured to handle acquisition, encoding, and display of the real-time stereoscopic data;
a head-mounted display (HMD) application configured to deliver an immersive real-time 3D video experience; and
a camera calibration tool set configured to perform 3D calibration for accurate depth recovery, the camera calibration tool set being executable either in free space or with the camera mounted on the first optical device, thereby providing operational flexibility across various clinical configurations.
7 . The system according to claim 6 , the stream client comprising:
a camera interface configured to acquire the real-time stereoscopic data;
a video encoder configured to compress the acquired real-time stereoscopic data; and
a real-time communication protocol client configured to efficiently handle the real-time stereoscopic data from acquisition through transmission or storage.
8 . The system according to claim 6 , the HM application comprising:
a 3D video player configured to receive the real-time stereoscopic data via a real-time streaming interface and to support playback of recorded videos with privacy controls including unique user authentication, data encryption, and audit logging protocols, the 3D video player configured to provide video enhancement features including adjustments for color, brightness, and contrast, as well as frame capture and digital magnification capabilities; and
a diagnosis auxiliary toolkit configured to utilize two-dimensional (2D) image data and 3D stereoscopic data representing anatomical features of the ocular region to assist in diagnosis, the diagnosis auxiliary toolkit comprising a depth measurement module and an artificial intelligence (AI)-assisted module for ocular abnormality detection.
9 . The system according to claim 1 , the streaming application comprising:
a real-time communication interface configured to minimize latency for real-time applications including remote diagnosis;
a backup streaming protocol configured to provide fallback streaming capability with broader compatibility; and
a secure media transport protocol configured to encrypt the real-time stereoscopic data to ensure a high standard of data security for the protection of Protected Health Information (PHI), thereby facilitating compliance with Health Insurance Portability and Accountability Act (HIPAA).
10 . A method for generating a Mixed Reality (MR)-based optical device, the method comprising:
a) a plurality of hardware modules integrated with a first optical device and configured to capture real-time stereoscopic data through an optical path, the real-time stereoscopic data comprising both image data and video data, captured via the first optical device;
b) a plurality of MR display devices operatively coupled to the plurality of hardware modules and configured to render the real-time stereoscopic data in an immersive environment; and
c) a plurality of software modules configured to acquire, process, and transmit the real-time stereoscopic data to the plurality of MR display devices,
the first optical device comprising at least one of a slit lamp and a microscope,
the MR-based optical device comprising at least one of an MR-based slit lamp and an MR-based microscope,
the plurality of software modules comprising:
i) a local application configured to support clinically intuitive interaction between a local operator and a patient, concurrently producing an enhanced, geometrically accurate three-dimensional (3D) visualization of an ocular region under examination;
ii) a streaming application configured to allow transmission of the real-time stereoscopic data to remote physicians equipped with MR, augmented reality (AR), or virtual reality (VR) headsets, thereby enabling collaborative clinical examinations across distributed locations; and
iii) a remote application configured to interface with one or more of the plurality of MR display devices, thereby enabling reliable data acquisition and interactive functionality, and
the plurality of software modules further configured to record and store the real-time stereoscopic data for on-demand review, thereby enhancing telemedicine capabilities in ophthalmology and improving emergency response in ophthalmic clinics.
11 . The method according to claim 10 , the plurality of hardware modules comprising:
mounting hardware structurally connected to the first optical device configured to support integration of cameras across multiple models of the first optical device; and
communication hardware configured to enable bidirectional data exchange with the plurality of MR display devices, thereby establishing a flexible platform interoperable with existing clinical optical device systems,
the clinical optical device systems comprising at least one of a clinical slit lamp system and a clinical microscope system.
12 . The method according to claim 11 , the mounting hardware comprising:
a structure configured to facilitate efficient attachment to and detachment from the first optical device;
a locking component configured to secure and maintain mechanical stability; and
fine adjustment assemblies configured to align the cameras with respect to image centering, focal distance, and rotational orientation,
the mounting hardware configured to:
be compatible with both portable handheld and table-mounted models;
achieve optimal alignment between optical axes of eyepieces of the first optical device and the cameras; and
eliminate artifacts including vignetting, distortion, or partial image loss.
13 . The method according to claim 11 , the cameras comprising a dual-channel video camera setup including two red, green, and blue (RGB) cameras configured to capture and stream the real-time stereoscopic data synchronously for immersive viewing,
each camera being mounted onto the first optical device, and
each camera being configured to preserve image resolution and depth perception consistent with the inherent optical characteristics of the first optical device.
14 . The method according to claim 11 , the communication hardware comprising:
integrated microphones and speakers within the plurality of MR display devices, configured to facilitate direct audio communication without the use of external devices; and
communication components configured to optimize real-time data exchange by maintaining a streaming latency of less than 0.1 seconds between the plurality of MR display devices, a local computing device, and a remote streaming server.
15 . The method according to claim 10 , the local application comprising:
a stream client configured to handle acquisition, encoding, and display of the real-time stereoscopic data;
a head-mounted display (HMD) application configured to deliver an immersive real-time 3D video experience; and
a camera calibration tool set configured to perform 3D calibration for accurate depth recovery, the camera calibration tool set being executable either in free space or with the camera mounted on the first optical device, thereby providing operational flexibility across various clinical configurations.
16 . The method according to claim 15 , the stream client comprising:
a camera interface configured to acquire the real-time stereoscopic data;
a video encoder configured to compress the acquired real-time stereoscopic data; and
a real-time communication protocol client configured to efficiently handle the real-time stereoscopic data from acquisition through transmission or storage, and
the HMD application comprising:
a 3D video player configured to receive the real-time stereoscopic data via a real-time streaming interface and to support playback of recorded videos with privacy controls including unique user authentication, data encryption, and audit logging protocols, the 3D video player configured to provide video enhancement features including adjustments for color, brightness, and contrast, as well as frame capture and digital magnification capabilities; and
a diagnosis auxiliary toolkit configured to utilize two-dimensional (2D) image data and 3D stereoscopic data representing anatomical features of the ocular region to assist in diagnosis, the diagnosis auxiliary toolkit comprising a depth measurement module and an artificial intelligence (AI)-assisted module for ocular abnormality detection.
17 . The method according to claim 10 , the streaming application comprising:
a real-time communication interface configured to minimize latency for real-time applications including remote diagnosis;
a backup streaming protocol configured to provide fallback streaming capability with broader compatibility; and
a secure media transport protocol configured to encrypt the real-time stereoscopic data to ensure a high standard of data security for the protection of Protected Health Information (PHI), thereby facilitating compliance with Health Insurance Portability and Accountability Act (HIPAA).
18 . A system for generating a Mixed Reality (MR)-based optical device, the system comprising:
a) a plurality of hardware modules integrated with a first optical device and configured to capture real-time stereoscopic data through an optical path, the real-time stereoscopic data comprising both image data and video data, captured via the first optical device;
b) a plurality of MR display devices operatively coupled to the plurality of hardware modules and configured to render the real-time stereoscopic data in an immersive environment; and
c) a plurality of software modules configured to acquire, process, and transmit the real-time stereoscopic data to the plurality of local MR display devices,
the plurality of hardware modules comprising:
mounting hardware structurally connected to the first optical device configured to support integration of cameras across multiple models of the first optical device; and
communication hardware configured to enable bidirectional data exchange with the plurality of MR display devices, thereby establishing a flexible platform interoperable with existing clinical optical device systems,
the mounting hardware comprising:
a structure configured to facilitate efficient attachment to and detachment from the first optical device;
a locking component configured to secure and maintain mechanical stability; and
fine adjustment assemblies configured to align the cameras with respect to image centering, focal distance, and rotational orientation,
the mounting hardware configured to:
be compatible with both portable handheld and table-mounted models;
achieve optimal alignment between optical axes of eyepieces of the first optical device and the cameras; and
eliminate artifacts including vignetting, distortion, or partial image loss,
the cameras comprising a dual-channel video camera setup including two red, green, and blue (RGB) cameras configured to capture and stream the real-time stereoscopic data synchronously for immersive viewing, each camera being mounted onto the first optical device, and each camera being configured to preserve image resolution and depth perception consistent with the inherent optical characteristics of the first optical device,
the communication hardware comprising:
integrated microphones and speakers within the plurality of MR display devices, configured to facilitate direct audio communication without the use of external devices; and
communication components configured to optimize real-time data exchange by maintaining a streaming latency of less than 0.1 seconds between the plurality of MR display devices, a local computing device, and a remote streaming server,
the plurality of software modules comprising:
i) a local application configured to support clinically intuitive interaction between a local operator and a patient, concurrently producing an enhanced, geometrically accurate three-dimensional (3D) visualization of an ocular region under examination;
ii) a streaming application configured to allow transmission of the real-time stereoscopic data to remote physicians equipped with MR, augmented reality (AR), or virtual reality (VR) headsets, thereby enabling collaborative clinical examinations across distributed locations; and
iii) a remote application configured to interface with one or more of the plurality of MR display devices, thereby enabling reliable data acquisition and interactive functionality,
the plurality of software modules further configured to record and store the real-time stereoscopic data for on-demand review, thereby enhancing telemedicine capabilities in ophthalmology and improving emergency response in ophthalmic clinics,
the local application comprising:
a stream client configured to handle acquisition, encoding, and display of the real-time stereoscopic data;
a head-mounted display (HMD) application configured to deliver an immersive real-time 3D video experience; and
a camera calibration tool set configured to perform 3D calibration for accurate depth recovery, the camera calibration tool set being executable either in free space or with the camera mounted on the first optical device, thereby providing operational flexibility across various clinical configurations,
the stream client comprising:
a camera interface configured to acquire the real-time stereoscopic data;
a video encoder configured to compress the acquired real-time stereoscopic data; and
a real-time communication protocol client configured to efficiently handle the real-time stereoscopic data from acquisition through transmission or storage,
the HMD application comprising:
a 3D video player configured to receive the real-time stereoscopic data via a real-time streaming interface and to support playback of recorded videos with privacy controls including unique user authentication, data encryption, and audit logging protocols;
the 3D video player configured to provide video enhancement features including adjustments for color, brightness, and contrast, as well as frame capture and digital magnification capabilities; and
a diagnosis auxiliary toolkit configured to utilize two-dimensional (2D) image data and 3D stereoscopic data representing anatomical features of the ocular region to assist in diagnosis, the diagnosis auxiliary toolkit comprising a depth measurement module and an artificial intelligence (AI)-assisted module for ocular abnormality detection,
the streaming application comprising:
a real-time communication interface configured to minimize latency for real-time applications including remote diagnosis;
a backup streaming protocol configured to provide fallback streaming capability with broader compatibility; and
a secure media transport protocol configured to encrypt the real-time stereoscopic data to ensure a high standard of data security for the protection of Protected Health Information (PHI), thereby facilitating compliance with Health Insurance Portability and Accountability Act (HIPAA),
the first optical device comprising at least one of a slit lamp and a microscope,
the MR-based optical device comprising at least one of an MR-based slit lamp and an MR-based microscope, and
the clinical optical device systems comprising at least one of a clinical slit lamp system and a clinical microscope system.