Line based image processing and flexible memory system
View Patent ↗Techniques are provided to implement line based processing of thermal images and a flexible memory system. In one example, individual lines of a thermal image frame may be provided to an image processing pipeline. Image processing operations may be performed on the individual lines in stages of the image processing pipeline. A memory system may be used to buffer the individual lines in the pipeline stages. In another example, a memory system may be used to send and receive data between various components without relying on a single shared bus. Data transfers may be performed between different components and different memories of the memory system using a switch fabric to route data over different buses. In another example, a memory system may support data transfers using different clocks of various components, without requiring the components and the memory system to all be synchronized to the same clock source.
1. A method comprising:
receiving a thermal image frame comprising a plurality of individual lines, wherein each individual line comprises substantially an entire row or column of thermal image data captured by a plurality of infrared sensors;
providing each individual line of the thermal image frame to a line based image processing pipeline;
performing image processing operations on the individual lines in stages of the image processing pipeline; and
buffering the individual lines for the pipeline stages, wherein the buffering comprises, for each individual line:
passing the individual line from a first one of the pipeline stages to a memory system,
storing the individual line in one of a plurality of virtual line buffers of the memory system, and
passing the stored individual line from the one of the virtual line buffers to a second one of the pipeline stages.
2. The method of claim 1 , wherein the image processing operations comprise:
storing a plurality of factory calibration terms associated with a single one of the individual lines in a memory buffer; and
applying the stored factory calibration terms to the one of the individual lines.
3. The method of claim 1 , wherein the thermal image frame is a current thermal image frame, wherein the image processing operations comprise:
buffering a first set of the individual lines comprising a subset of the current image frame in a first set of corresponding memory buffers of a memory system;
buffering a second set of individual lines comprising a subset of a previous thermal image frame in a second set of corresponding memory buffers of the memory system;
comparing the first and second sets of individual lines to determine if a scene has changed; and
if the scene has changed, accumulating at least one of the image frames in an accumulation buffer of the memory system to provide a temporally filtered image frame.
4. The method of claim 1 , wherein the image processing operations comprise:
buffering a set of the individual lines comprising a subset of the image frame in a first set of corresponding memory buffers;
buffering a subset of a pixel map in a second set of corresponding memory buffers;
replacing one or more pixels of the individual lines based on the subset of the pixel map;
applying a plurality of distortion correction terms to one of the individual lines; and
applying look up table (LUT) data to the one of the individual lines.
5. The method of claim 1 , wherein the thermal image frame is a current thermal image frame, wherein the image processing operations comprise:
buffering a set of individual lines comprising a subset of a previous image frame in a set of corresponding memory buffers; and
processing the individual lines of the previous thermal image frame to determine row and column noise terms for the current thermal image frame.
6. The method of claim 1 , wherein the thermal image frame is an unblurred thermal image frame, wherein the image processing operations comprise:
determining a plurality of non-uniform correction (NUC) terms for a single one of the individual lines of the unblurred thermal image frame from an intentionally blurred thermal image frame;
storing the NUC terms associated with a single one of the individual lines in a memory buffer; and
applying the stored NUC terms to the one of the individual lines.
7. The method of claim 1 , wherein each individual line comprises approximately 80 pixels or fewer.
8. The method of claim 1 , further comprising:
operating a switch fabric to couple a first port of a memory buffer with a memory block of the memory buffer, wherein the memory block comprises a single interface adapted to support a single read or write operation at a time;
transferring a first one of the individual lines between a first component and the memory block over a first bus, through the first port, and through the single interface;
subsequently operating the switch fabric to couple a second port of the memory buffer with the memory block; and
transferring a second one of the individual lines between a second component and the memory block over a second bus, through the second port, and through the single interface.
9. An infrared imaging module comprising:
a sensor input block adapted to receive a thermal image frame comprising a plurality of individual lines, wherein each individual line comprises substantially an entire row or column of thermal image data captured by a plurality of infrared sensors;
a processing device comprising a line based image processing pipeline adapted to perform image processing operations on the individual lines in stages of the image processing pipeline;
a memory system adapted to buffer the individual lines for the pipeline stages;
wherein the processing device is adapted to, for each individual line, pass the individual line from a first one of the pipeline stages to a memory system; and
wherein the memory system is adapted to store the individual line in one of a plurality of virtual line buffers of the memory system and pass the stored individual line from the one of the virtual line buffers to a second one of the pipeline stages.
10. The infrared imaging module of claim 9 , wherein the processing device is adapted to:
store factory calibration terms associated with a single one of the individual lines in a memory buffer of the memory system; and
apply the stored factory calibration terms to the one of the individual lines.
11. The infrared imaging module of claim 9 , wherein the thermal image frame is a current thermal image frame,
wherein the memory system is adapted to:
buffer a first set of the individual lines comprising a subset of the current image frame in a first set of corresponding memory buffers of the memory system, and
buffer a second set of individual lines comprising a subset of a previous thermal image frame in a second set of corresponding memory buffers of the memory system; and
wherein the processing device is adapted to:
compare the first and second sets of individual lines to determine if a scene has changed, and
if the scene has changed, accumulate at least one of the image frames in an accumulation buffer of the memory system to provide a temporally filtered image frame.
12. The infrared imaging module of claim 9 ,
wherein the memory system is adapted to:
buffer a set of the individual lines comprising a subset of the image frame in a first set of corresponding memory buffers of the memory system, and
buffer a subset of a pixel map in a second set of memory buffers of the memory system; and
wherein the processing device is adapted to:
replace one or more pixels of the first set of individual lines based on the subset of a pixel map,
apply a plurality of distortion correction terms to one of the individual lines, and
apply look up table (LUT) data to the one of the individual lines.
13. The infrared imaging module of claim 9 , wherein the thermal image frame is a current thermal image frame,
wherein the memory system is adapted to buffer a set of individual lines comprising a subset of a previous image frame in a set of corresponding memory buffers of the memory system; and
wherein the processing device is adapted to process the individual lines of the previous thermal image frame to determine row and column noise terms for the current thermal image frame.
14. The infrared imaging module of claim 9 , wherein the thermal image frame is an unblurred thermal image frame, wherein the processing device is adapted to:
determine a plurality of non-uniform correction (NUC) terms for a single one of the individual lines of the unblurred thermal image frame from an intentionally blurred thermal image frame;
store the NUC terms in a memory buffer of the memory system; and
apply the stored NUC terms to the one of the individual lines.
15. The infrared imaging module of claim 9 , wherein each individual line comprises approximately 80 pixels or fewer.
16. The infrared imaging module of claim 9 , wherein the memory system is adapted to:
operate a switch fabric to couple a first port of a memory buffer with a memory block of the memory buffer, wherein the memory block comprises a single interface adapted to support a single read or write operation at a time;
transfer a first one of the individual lines between a first component and the memory block over a first bus, through the first port, and through the single interface;
subsequently operate the switch fabric to couple a second port of the memory buffer with the memory block; and
transfer a second one of the individual lines between a second component and the memory block over a second bus, through the second port, and through the single interface.