Systems, methods, and media for high dynamic range imaging using single-photon and conventional image sensor data
In accordance with some embodiments, systems, methods, and media for high dynamic range imaging using single-photon and conventional image sensor data are provided. In some embodiments, the system comprises: first detectors configured to detect a level of photons proportional to incident photon flux; second detectors configured to detect arrival of individual photons; a processor programmed to: receive, from the first detectors, first values indicative of photon flux from a scene with a first resolution; receive, from the second detectors, second values indicative of photon flux from the scene with a lower resolution; provide a first encoder of a trained machine learning model first flux values based on the first values, provide the second encoder of the model second flux values; receive, as output, values indicative of photon flux from the scene; and generate a high dynamic range image based on the third plurality of values.
1. A system for generating high dynamic range digital images, comprising:
a communication connection configured to receive readout data from an image data source, the image data source comprising at least one image sensor;
a processor; and
at least one memory connected to receive and store the readout data received by the communication connection, and having stored thereon a set of software instructions which, when executed by the processor, cause the processor to:
receive, via the communication connection, first readout data for a scene, the first readout data indicative of flux detected by a first detector of the at least one image sensor having a first dynamic range and a first resolution;
receive, via the communication connection, second readout data for the scene, the second readout data indicative of flux at a second detector of the image sensor, the second detector having a second dynamic range that is lower than the first dynamic range and a second resolution that is higher than the first resolution;
provide the first readout data and the second readout data as inputs to a trained machine learning model, wherein the trained machine learning model was trained using a dataset comprising image data for scenes with corresponding estimated and known flux values from sensors having different dynamic ranges;
receive, as output from the trained machine learning model, third readout data based on properties of both the first readout data and the second readout data; and
generate an image corresponding to the scene, wherein the image has a third resolution that is higher than the first resolution and a third dynamic range that is higher than the second dynamic range.
2. The system of claim 1 , wherein the second detector comprises a plurality of second detectors which capture color and brightness information for a scene.
3. The system of claim 2 , wherein the plurality of second detectors comprises a complementary semiconductor metal oxide (CMOS) pixel array.
4. The system of claim 1 , wherein the first detector comprises at least one single photon detector.
5. The system of claim 3 , wherein the first detector comprises at least one single-photon avalanche diode (SPAD) detector, and the flux detected by the first detector is based on detection of individual photons by the SPAD detector during a given time period.
6. The system of claim 1 , wherein the trained machine learning model includes a first skip connection between a layer of a first encoder, configured to process the first readout data indicative of flux detected by the first detector, and a layer of a decoder, and a second skip connection between a layer of a second encoder, configured to process the second readout data indicative of flux detected by the second detector, and the layer of the decoder, wherein the trained machine learning model is configured to concatenate values from the layer of the first encoder and values from the layer of the second encoder.
7. The system of claim 1 , wherein the processor is further programmed to:
estimate a first plurality of flux values associated with the first readout data using a relationship:
Φ
^
CMOS
=
N
^
T
CMOS
q
CMOS
T
,
Where {circumflex over (ϕ)} CMOS is the estimated flux for a portion of the scene, {circumflex over (N)} T CMOS is a value output by the first detector, q CMOS is a sensitivity of the first detector, and T is exposure time; and
estimate a second plurality of flux values associated with the second readout data using a relationship:
Φ
^
SPC
=
N
^
T
SPC
SPC
/
q
SPAD
T
SPC
-
τ
d
n
^
τ
SPC
SPC
,
where {circumflex over (ϕ)} SPC is the estimated flux for the portion of the scene, T SPC is exposure time, {circumflex over (N)} T SPC SPC is a photon count corresponding to a number of photon detections in exposure time T SPC , q SPAD is a sensitivity of the detector, and Ta is a dead time of the detector.
8. A method for generating high dynamic range digital images, the method comprising:
receiving a first plurality of image data for a scene;
determining a first plurality of photon flux indications;
receiving a second plurality of image data for the scene;
determining a second plurality of photon flux indications;
providing, as an input to a first encoder of a trained machine learning model, the first plurality of photon flux indications;
concatenating, using the trained machine learning model, values from a first encoder layer and values from a second encoder layer;
receiving, as an output from the trained machine learning model, a third plurality of photon flux indications for the scene; and
generating a high dynamic range image based on the third plurality of photon flux indications.
9. The method of claim 8 , wherein the output is an HDR output.
10. The method of claim 8 , wherein concatenating values from a first encoder layer and values from a second encoder layer further comprises introducing a plurality of attention gates.