Comprehensive fixed pattern noise cancellation
The disclosure extends to methods, systems, and computer program products for producing an image in light deficient environments having cancelled fixed pattern noise.
1. A system comprising:
an image sensor comprising a pixel array, wherein the pixel array comprises a plurality of pixels configured to accumulate electromagnetic radiation and an integration time of the pixel array wherein the pixel array accumulates the electromagnetic radiation; and
an emitter that emits a plurality of pulses of electromagnetic radiation, wherein the pixel array is actuated at a sensing interval that corresponds to a pulse interval of the emitter;
wherein the image sensor comprises a blanking period;
wherein the image sensor comprises a readout period during which the plurality of pixels read out an electrical charge or current based on an amount of electromagnetic radiation accumulated by each of the plurality of pixels;
wherein the image sensor reads out dark frame data during a dark frame readout period in response to the emitter not pulsing electromagnetic radiation during a preceding blanking period;
wherein the integration time is constant or variable on a frame-to-frame basis according to an auto-exposure algorithm; and
wherein the system obtains pixel calibration data from the dark frame data based on the integration time.
2. The system of claim 1 , further comprising:
a processor in communication with the image sensor and the emitter; and
memory operably coupled to the processor, wherein the memory stores instructions executable by the processor, and wherein the instructions comprise:
instructing the image sensor to operate according to a sensor operational cycle, wherein the sensor operational cycle comprises the blanking period for accumulating the electromagnetic radiation and the readout period for reading out the electrical charge or the current; and
instructing the emitter to emit the plurality of pulses of electromagnetic radiation according to a pulse cycle;
wherein the pulse cycle corresponds with the sensor operational cycle such that the emitter emits the plurality of pulses of electromagnetic radiation during the blanking period of the sensor operational cycle.
3. The system of claim 2 , wherein the pulse cycle comprises a dark emission period, and wherein the dark emission period is one or more dark emission periods wherein the emitter does not emit electromagnetic radiation.
4. The system of claim 3 , wherein the one or more dark emission periods precedes the dark frame readout period, such that the image sensor reads out the dark frame data subsequent to the emitter not emitting electromagnetic radiation.
5. The method of claim 4 , wherein the pulse cycle is a pattern repeated by the emitter, and wherein the pulse cycle comprises:
one or more visible emission periods wherein the emitter pulses a visible wavelength of electromagnetic radiation; and
the dark emission period wherein the emitter does not emit electromagnetic radiation.
6. The system of claim 5 , wherein the dark emission period is repeated at regular intervals such that the image sensor reads out the dark frame data during a plurality of different readout periods to generate a plurality of dark frames.
7. The system of claim 6 , wherein at least a portion of the plurality of dark frames are input into a running average process to calculate a pixel offset correction.
8. The system of claim 7 , wherein the pixel offset correction is subtracted from non-dark frame data output by the pixel array to suppress fixed pattern noise in the non-dark frame data.
9. The system of claim 7 , wherein the non-dark frame data comprises one or more of:
a red frame read out by the pixel array subsequent to a red emission period;
a green frame read out by the pixel array subsequent to a green emission period; or
a blue frame read out by the pixel array subsequent to a blue emission period.
10. The system of claim 2 , wherein the instructions further comprise instructing the emitter and the image sensor to operate according to an operation interval, and wherein the operational interval comprises the blanking period and the readout period of the image sensor.
11. The system of claim 1 , further comprising a processor configured to calibrate the pixel array based on the calibration data obtained from the dark frame data.
12. The system of claim 10 , wherein the pixel array is temperature dependent such a plurality of dark frame data readouts is used to calibrate the pixel array over time.
13. The system of claim 12 , wherein the plurality of dark frame data readouts are output by the image sensor at regular intervals and according to a pulse cycle.
14. The system of claim 13 , wherein at least a portion of the plurality of dark frame data readouts are averaged using simple exponential smoothing, and wherein the averaged dark frame data is used to suppress fixed pattern noise in image data output by the image sensor.
15. The system of claim 1 , wherein at least a portion of the plurality of pixels of the pixel array is optical black pixels comprising light shielding.
16. The system of claim 15 , wherein a quantity of optical black pixels in the pixel array is minimized such that the pixel array comprises only a first column of optical black pixels disposed on a first end of the pixel array, and a second column of optical black pixels disposed on an opposite end of the pixel array relative to the first column.
17. The system of claim 1 , wherein a pixel offset for calibrating the pixel array is linearly dependent on the integration time for the pixel array, and wherein the pixel calibration data for a given integration time is computed using linear interpolation based on the dark frame data.
18. The system of claim 1 , wherein the dark frame data is subtracted from non-dark frame data output by the image sensor to suppress fixed pattern noise.
19. The system of claim 18 , wherein the non-dark frame data comprises data read out by the pixel array subsequent to the emitter pulsing electromagnetic radiation.
20. The system of claim 1 , wherein the system further comprises a dark emission period that is concurrent with the blanking period of the image sensor.