Systems and methods for low-light image enhancement
A method for enhancing low brightness regions in endoscopic video includes receiving a sequence of video frames captured during an endoscopic procedure. A brightness map is generated for each frame of the sequence of video frames. The brightness map is generated based on smoothing regional brightness in the respective frame based on regional brightness gradients. The brightness map for the respective frame includes scaling weights corresponding to pixels of the respective frame. An enhanced sequence of video frames is generated in real-time by adjusting pixel values of each frame of the sequence of video frames by scaling the pixel values of the respective frame by the scaling weights of the brightness map for the respective frame to enhance brightness of low brightness regions of the respective frame. The enhanced sequence of video frames is displayed in real-time.
1. A method for enhancing low brightness regions in endoscopic video comprising:
receiving a sequence of video frames captured during an endoscopic procedure;
generating a brightness map for each frame of the sequence of video frames, wherein the brightness map is generated based on smoothing regional brightness in the respective frame based on regional brightness gradients, and wherein the brightness map for the respective frame comprises scaling weights corresponding to pixels of the respective frame;
generating in real-time an enhanced sequence of video frames as the sequence of video frames is received during the endoscopic procedure by adjusting pixel values of each frame of the sequence of video frames by scaling the pixel values of the respective frame by the scaling weights of the brightness map for the respective frame to enhance brightness of low brightness regions of the respective frame; and
displaying the enhanced sequence of video frames to a surgeon in real-time during the endoscopic procedure.
2. The method of claim 1 , wherein the enhanced sequence of video frames is generated in real-time by a graphics processing unit (GPU).
3. The method of claim 1 , comprising down-sampling the respective frame and generating the respective brightness map based on the down-sampling of the respective frame.
4. The method of claim 3 , comprising up-sampling the respective brightness map and adjusting the pixel values of the respective frame based on the up-sampling of the respective brightness map.
5. The method of claim 4 , wherein the respective brightness map is up-sampled using bilinear interpolation.
6. The method of claim 1 , wherein generating the brightness map for each frame of the sequence of video frames comprises minimizing a cost function, wherein the cost function comprises a sparse matrix and minimizing the cost function comprises constructing a compressed sparse matrix structure.
7. The method of claim 6 , wherein the compressed sparse matrix structure is constructed in compressed sparse row format.
8. The method of claim 6 , wherein the compressed sparse matrix structure includes a row index vector, a column index vector, and a non-zero values vector, the row index vector and column index vector are the same for each frame, and constructing the sparse matrix structure comprises updating the non-zero values vector for the respective frame.
9. The method of claim 6 , comprising using an iterative conjugate gradient method to solve the minimization of the cost function.
10. The method of claim 1 , wherein each of the scaling weights is computed based on a brightness gradient of a corresponding region of pixels of the respective frame.
11. The method of claim 10 , wherein adjusting the pixel values of each of the frames comprises scaling the pixel values by the weights of the matrix of weights.
12. The method of claim 10 , wherein each of the frames comprises different color channels and adjusting pixel values of each of the frames comprises scaling each of the color channels of a given pixel by the same weight.
13. The method of claim 1 , comprising, prior to generating a brightness map for each frame of the sequence of video frames: generating an initial brightness map estimate for each frame of the sequence of video frames; and down-sampling the initial brightness map estimate for each frame of the sequence of video frames.
14. The method of claim 13 , wherein the brightness map for each frame of the sequence of video frames is generated based on the respective down-sampled initial brightness map estimate.
15. A system comprising one or more processors, memory, and one or more programs stored in the memory for execution by the one or more processors for:
receiving a sequence of video frames captured during an endoscopic procedure;
generating a brightness map for each frame of the sequence of video frames, wherein the brightness map is generated based on smoothing regional brightness in the respective frame based on regional brightness gradients, and wherein the brightness map for the respective frame comprises scaling weights corresponding to pixels of the respective frame;
generating in real-time an enhanced sequence of video frames as the sequence of video frames is received during the endoscopic procedure by adjusting pixel values of each frame of the sequence of video frames by scaling the pixel values of the respective frame by the scaling weights of the brightness map for the respective frame to enhance brightness of low brightness regions of the respective frame; and
displaying the enhanced sequence of video frames to a surgeon in real-time during the endoscopic procedure.
16. The system of claim 15 , wherein the enhanced sequence of video frames is generated in real-time by a graphics processing unit (GPU).
17. The system of claim 15 , wherein the one or more programs include instructions for down-sampling the respective frame and generating the respective brightness map based on the down-sampling of the respective frame.
18. The system of claim 17 , wherein the one or more programs include instructions for up-sampling the respective brightness map and adjusting the pixel values of the respective frame based on the up-sampling of the respective brightness map.
19. The system of claim 18 , wherein the respective brightness map is up-sampled using bilinear interpolation.
20. The system of claim 15 , wherein generating the brightness map for each frame of the sequence of video frames comprises minimizing a cost function, wherein the cost function comprises a sparse matrix and minimizing the cost function comprises constructing a compressed sparse matrix structure.
21. The system of claim 20 , wherein the compressed sparse matrix structure is constructed in compressed sparse row format.
22. The system of claim 20 , wherein the compressed sparse matrix structure includes a row index vector, a column index vector, and a non-zero values vector, the row index vector and column index vector are the same for each frame, and constructing the sparse matrix structure comprises updating the non-zero values vector for the respective frame.
23. The system of claim 20 , wherein the minimization of the cost function is solved via an iterative conjugate gradient method.
24. The system of claim 15 , wherein each of the scaling weights is computed based on a brightness gradient of a corresponding region of pixels of the respective frame.
25. The system of claim 24 , wherein adjusting the pixel values of each of the frames comprises scaling the pixel values by the weights of the matrix of weights.
26. The system of claim 24 , wherein each of the frames comprises different color channels and adjusting pixel values of each of the frames comprises scaling each of the color channels of a given pixel by the same weight.
27. The system of claim 15 , wherein the one or more programs include instructions for, prior to generating a brightness map for each frame of the sequence of video frames: generating an initial brightness map estimate for each frame of the sequence of video frames; and down-sampling the initial brightness map estimate for each frame of the sequence of video frames.
28. The system of claim 27 , wherein the brightness map for each frame of the sequence of video frames is generated based on the respective down-sampled initial brightness map estimate.
29. A non-transitory computer readable medium comprising instructions for execution by one or more processors of a system to cause the system to:
receive a sequence of video frames captured during an endoscopic procedure;
generate a brightness map for each frame of the sequence of video frames, wherein the brightness map is generated based on smoothing regional brightness in the respective frame based on regional brightness gradients, and wherein the brightness map for the respective frame comprises scaling weights corresponding to pixels of the respective frame;
generate in real-time an enhanced sequence of video frames as the sequence of video frames is received during the endoscopic procedure by adjusting pixel values of each frame of the sequence of video frames by scaling the pixel values of the respective frame by the scaling weights of the brightness map for the respective frame to enhance brightness of low brightness regions of the respective frame; and
display the enhanced sequence of video frames to a surgeon in real-time during the endoscopic procedure.