Optical image stabilization and active alignment of sensors in endoscopic camera systems and methods of using the same
Systems and methods are provided that enable optical image stabilization and image sensor alignments. By enabling image sensors to be adjusted based on identified misalignments, users of the systems and methods disclosed herein can adjust imaging during the course of a surgery or surgical procedure, without needing to stop the surgery or surgical procedure to manually realign optical components. The optical image stabilization allows for adjustment of sensor components to compensate for shifts or other movement of imaging devices and to maintain focus on a target object. The optical image stabilization also includes digital masks that can be overlaid on an image to block distracting mask shifting.
1. An apparatus, comprising:
an endoscope, the endoscope comprising:
an optical component configured to split light into two or more light paths;
a first sensor disposed on a chassis and aligned with a first light path of the two or more light paths; and
a second sensor disposed proximate the optical component and aligned with a second light path of the two or more light paths, wherein the first sensor is configured to move relative to at least one of the optical component and the second sensor, and wherein the movement is orthogonal to an optical axis of the optical component; and
a processor,
wherein the first sensor generates a first video stream, and wherein data stored on a memory cause the processor to:
receive information associated with a movement of the endoscope;
determine, based on the information, a movement of the first sensor to compensate for the movement of the endoscope;
determine, based on the movement of the endoscope, a change in position of an object depicted in the first video stream;
shift, based on the movement of the endoscope, a digital mask; and
overlay, on the video stream, the digital mask.
2. The apparatus of claim 1 , wherein the memory, when processed by the processor, causes the processor to:
determine a misalignment between the first sensor and the second sensor;
determine, based on a registration between the first sensor and the second sensor, a first change in pose of the first sensor to compensate for misalignment; and
cause the first change in pose of the first sensor.
3. The apparatus of claim 2 , wherein causing the first change in pose of the first sensor includes causing the first sensor to move orthogonal to the optical axis of the optical component.
4. The apparatus of claim 2 , wherein determining the misalignment includes:
receiving the first video stream from the first sensor and a second video stream from the second sensor; and
performing feature detection on the first video stream and on the second video stream.
5. The apparatus of claim 4 , wherein the processor is further caused to:
determine, based on the feature detection, an amount of offset between the first video stream and the second video stream.
6. The apparatus of claim 5 , wherein determining the first change in pose of the first sensor to compensate for the misalignment is based at least in part on the amount of offset between the first video stream and the second video stream.
7. The apparatus of claim 1 , wherein the digital mask is smaller than an endoscopic mask, the endoscopic mask being an artifact of one or more optical elements of an additional attached endoscope or the apparatus, and wherein a rendering of the video stream on a display does not change when the endoscope moves and the digital mask is overlaid on the video stream.
8. The apparatus of claim 1 , wherein at least one of the first sensor and the second sensor is not mounted to the optical component.