Super resolution and color motion artifact correction in a pulsed laser mapping imaging system

1. A method comprising:

actuating an emitter to emit a plurality of pulses of electromagnetic radiation;

sensing reflected electromagnetic radiation resulting from the plurality of pulses of electromagnetic radiation with a pixel array of an image sensor to generate a plurality of frames;

detecting motion across two or more sequential frames of the plurality of frames;

compensating for the detected motion; and

combining the two or more sequential frames;

wherein at least a portion of the plurality of frames comprises a laser mapping frame sensed in response to the emitter pulsing a laser mapping pattern, wherein the laser mapping frame comprises data for determining a measurement to an accuracy of less than 10 centimeters.

2. The method of claim 1 , wherein compensating for the detected motion comprises:

upscaling a first frame of the two or more sequential frames using interpolation to generate a first upscaled frame;

upscaling the first frame without using interpolation to generate a second upscaled frame, wherein the second upscaled frame comprises a first set of empty pixels; and

filling in the first set of empty pixels of the second upscaled frame with pixel data from the first upscaled frame.

3. The method of claim 2 , wherein compensating for the detected motion further comprises:

upscaling a second frame of the two or more sequential frames to generate a third upscaled frame; and

filling in a second set of empty pixels in the second upscaled frame with pixel data from the third upscaled frame.

4. The method of claim 1 , wherein the two or more sequential frames comprises a red exposure frame, a green exposure frame, and a blue exposure frame, and wherein the method comprises combining the two or more sequential frames to generate a Red Green Blue (“RGB”) image frame.

5. The method of claim 1 , wherein the two or more sequential frames comprises a luminance (Y) exposure frame, a chrominance blue (Cb) exposure frame, and a chrominance red (Cr) exposure frame, and wherein the method comprises combining the two or more sequential frames to generate a YCbCr image frame.

6. The method of claim 1 , wherein sensing the reflected electromagnetic radiation comprises:

generating a first frame based on a pulse of electromagnetic radiation of a first color partition;

generating a second frame based on a pulse of electromagnetic radiation of a second color partition; and

generating a third frame based on a pulse of electromagnetic radiation of the first color partition;

wherein the second frame is captured between the first frame and the third frame;

wherein detecting motion across the two or more sequential frames comprises calculating a relative motion estimate based on the first frame and the third frame using block matching; and

wherein compensating for the detected motion comprises generating a motion compensated frame for the second exposure frame based on the relative motion estimate.

7. The method of claim 6 , further comprising:

determining a first motion vector for the first frame and a second motion vector for the second frame; and

shifting a block of pixels in the first frame by the first motion vector.

8. The method of claim 1 , further comprising:

performing bilinear interpolation on luminance data in the two or more sequential frames to generate a first upscaled dataset;

performing bicubic interpolation on the luminance data to generate a second upscaled dataset; and

calculating a baseline with no interpolation of the luminance data to generate a third upscaled dataset.

9. The method of claim 8 , wherein detecting motion across two or more sequential frames comprises one or more of:

segmenting data sensed by the pixel array into segments of pixels and nearest neighboring frames;

shifting each segment of pixels in the x direction and comparing with a neighboring frame at a same resolution to identify motion of an object being imaged in the x direction;

shifting each segment of pixels in the x direction in sub-pixel increments and comparing to the first upscaled dataset to identify motion of the object being imaged in the x direction with increased precision;

shifting each segment of pixels in the y direction and comparing with a neighboring frame to identify motion of an object being imaged in the y direction; or

shifting each segment of pixels in the y direction in sub-pixel increments and comparing to the first upscaled dataset to identify motion of the object being imaged in the y direction with increased precision.

10. The method of claim 1 , wherein sensing the reflected electromagnetic radiation comprises sensing during a readout period of the pixel array, wherein the readout period is a duration of time when active pixels in the pixel array are read.

11. The method of claim 1 , wherein actuating the emitter comprises actuating the emitter to emit, during a pulse duration, a plurality of sub-pulses of electromagnetic radiation having a sub-duration shorter than the pulse duration.

12. The method of claim 1 , wherein actuating the emitter comprises actuating the emitter to emit two or more wavelengths simultaneously as a single pulse or a single sub-pulse.

13. The method of claim 1 , wherein actuating the emitter comprises actuating the emitter to pulse the laser mapping pattern at a duration and frequency such that the laser mapping pattern is not visible to a user viewing an output video stream.

14. The method of claim 1 , wherein sensing the reflected electromagnetic radiation comprises generating a fluorescence frame, and wherein the method further comprises providing the fluorescence frame to a corresponding fluorescence system that determines a location of a tissue structure within a scene based on the fluorescence frame.

15. The method of claim 14 , further comprising:

receiving the location of the tissue structure from the corresponding fluorescence system;

generating an overlay frame comprising the location of the tissue structure; and

combining the overlay frame with a color image frame depicting the scene to indicate the location of the tissue structure within the scene.

16. The method of claim 15 , further comprising:

providing the laser mapping frame to a corresponding laser mapping system that calculates a topographical map of the scene and/or a dimension of one or more objects within the scene;

provide the location of the tissue structure to the corresponding laser mapping system; and

receive one or more of a topographical map of the tissue structure or a dimension of the tissue structure from the corresponding laser mapping system.

17. The method of claim 16 , wherein the tissue structure comprises one or more of a nerve, a ureter, a blood vessel, an artery, a blood flow, or a tumor.

18. The method of claim 1 , further comprising synchronizing timing of the plurality of pulses of electromagnetic radiation to be emitted during a blanking period of the image sensor, wherein the blanking period corresponds to a time between a readout of a last row of active pixels in the pixel array and a beginning of a next subsequent readout of active pixels in the pixel array.

19. The method of claim 1 , wherein the two or more sequential frames are captured sequentially in time based on two or more pulses of electromagnetic radiation emitted by the emitter sequentially in time.

20. The method of claim 1 , wherein the laser mapping pattern comprises on or more of vertical hashing, horizontal hashing, a raster grid of discrete points, an occupancy grid map, or a dot array.

21. The method of claim 1 , wherein actuating the emitter comprises actuating the emitter to emit a sequence of pulses of electromagnetic radiation repeatedly sufficient for generating a video stream comprising a plurality of image frames, wherein at least a portion of the plurality of image frames comprise data from a plurality of frames comprising a color frame and the laser mapping frame.

22. The method of claim 1 , wherein the laser mapping frame comprises information for determining real time measurements comprising one or more of:

a distance from an endoscope to an object;

an angle between an endoscope and the object; or

surface topology information about the object.

23. The method of claim 1 , wherein the laser mapping frame comprises information for determining the measurement to an accuracy of less than one millimeter.

24. The method of claim 1 , wherein actuating the emitter to emit the plurality of pulses of electromagnetic radiation comprises actuating the emitter to emit a plurality of tool-specific laser mapping patterns for each of a plurality of tools within a scene.

25. The method of claim 1 , wherein the laser mapping pattern emitted by the emitter comprises a first output of electromagnetic radiation and a second output of electromagnetic radiation that are independent from one another, wherein the first output of electromagnetic radiation is for light illumination and the second output of electromagnetic radiation is for tool tracking.

26. The method of claim 1 , further comprising identifying a luminance plane in the two or more sequential frames, and wherein detecting the motion across the two or more sequential frames comprises detecting motion based on data in the luminance plane for the two or more sequential frames.