Systems, methods and devices for multispectral imaging and non-linear filtering of vector valued data
View Patent ↗An apparatus and method for multispectral imaging comprising an array of filters in a mosaic pattern, a sensor array and an acquisition and processing module. The sensor array being disposed to receive an image that has been filtered by the array of filters. The acquisition and processing module processes the output of the sensor array (or mosaic acquired data) to provide a processed image. The acquisition and processing module processes the mosaic acquired data by performing first interpolation on the mosaic acquired data by the sensor array to provide a first approximation and performing second interpolation on the values of the first approximation to provide a second approximation.
1. A multi-spectral imaging apparatus, comprising:
an array of filters in a mosaic pattern;
a sensor array disposed to receive an image that has been filtered by said array of filters; and
an acquisition and processing module disposed to receive the output of said sensor array and for processing said output of said sensor array to provide a processed image, wherein said acquisition and processing module is operable to:
perform a first interpolation on said output of said sensor array to provide a first approximation; and
perform a second interpolation on the values of said first approximation to provide a second approximation by approximating values at each pixel of said processed image by, for each given pixel in said processed image, interpolating over the values from said first approximation evaluated at those other pixels for which the values of said first approximation at said other pixels are close to the values at said given pixel.
2. The apparatus of claim 1 , wherein said sensor array comprises a plurality of pixels; wherein said first approximation comprises a vector of values comprising values corresponding to each of said filters for each pixel in said plurality of pixels; and wherein said acquisition and processing module is operable to perform said second interpolation on the values of said first approximation to provide said second approximation by approximating values at each pixel of said processed image by, for each given pixel in said processed image, and each given filter, interpolating over the values corresponding to said given filter from said first approximation evaluated at those other pixels for which said vector of values of said first approximation at said other pixels are close to said vector of values at said given pixel.
3. The apparatus of claim 2 , wherein said array of filters is an array of spectral filters.
4. The apparatus of claim 2 , wherein said acquisition and processing module is operable to detect and track explosives.
5. The apparatus of claim 2 , wherein said acquisition and processing module is operable to detect and track biohazards.
6. The apparatus of claim 2 , wherein said acquisition and processing module is operable to recognize target automatically.
7. The apparatus of claim 2 , wherein said acquisition and processing module is operable to perform said first interpolation by interpolating said output of sensor array bilinearly to provide an (n×m×L) multispectral data cube I(i, j, k), where i and j are spatial indices and k is spectral index.
8. The apparatus of claim 2 , wherein said acquisition and processing module is operable to perform said second interpolation by:
forming a matrix
M
(
i
1
,
j
1
,
i
2
,
j
2
)
=
f
(
I
_
(
i
1
,
j
1
)
,
I
_
(
i
2
,
j
2
)
,
ɛ
)
∑
i
3
=
1
…
n
,
j
3
=
1
…
m
f
(
I
_
(
i
1
,
j
1
)
,
I
_
(
i
3
,
j
3
)
,
ɛ
)
,
where Ī(i, j) denotes the vector <I(i, j, 1), I(i, j, 2), . . . , I(i, j, L)>, with i and j fixed, k ranging over all integer values from 1 to L; and
calculating a reconstructed multispectral datacube
I
~
(
i
,
j
,
k
)
=
∑
i
1
=
1
…
n
,
j
1
=
1
…
m
M
(
i
,
j
,
i
1
,
j
1
)
·
I
(
i
1
,
j
1
,
k
)
for k ranging over all integer values from 1 to L.
9. The apparatus of claim 8 , wherein said acquisition and processing module is operable to perform said second interpolation by forming a matrix:
M
(
i
1
,
j
1
,
i
2
,
j
2
)
=
ⅇ
-
I
_
(
i
1
,
j
1
)
-
I
_
(
i
2
,
j
2
)
2
/
ɛ
∑
i
3
=
1
…
n
,
j
3
=
1
…
m
ⅇ
-
I
_
(
i
1
,
j
1
)
-
I
_
(
i
3
,
j
3
)
2
/
ɛ
,
where Ī(i, j) denotes the vector <I(i, j, 1), I(i, j, 2), . . . , I(i, j, L)>, with i and j fixed, k ranging over all integer values from 1 to L.
10. The apparatus of claim 9 , wherein said acquisition and processing module is operable to perform said second interpolation by calculating a reconstructed multispectral datacube:
I
~
~
(
i
,
j
,
k
)
=
∑
i
1
=
i
-
C
…
i
+
C
…
j
1
=
j
-
C
…
j
+
C
M
(
i
,
j
,
i
1
,
j
1
)
·
I
(
i
1
,
j
1
,
k
)
for a fixed square region centered at (i, j) and having a length of 2C+1.
11. A method of multi-spectral imaging, comprising the steps of:
receiving an image that has been filtered by an array of filters in a mosaic pattern by a sensor array; and
processing said output of said sensor array by an acquisition and processing module to provide a processed image by:
performing a first interpolation on said output of said sensor array to provide a first approximation; and
performing a second interpolation on the values of said first approximation to provide a second approximation by approximating values at each pixel of said processed image by, for each given pixel in said processed image, interpolating over the values from said first approximation evaluated at those other pixels for which the values of said first approximation at said other pixels are close to the values at said given pixel.
12. The method of claim 11 , wherein said sensor array comprises a plurality of pixels; and further comprising the steps of:
performing said first interpolation on said output of said sensor array to provide said first approximation comprising a vector of values comprising values corresponding to each of said filters for each pixel in said plurality of pixels; and
performing interpolation on the values of said first approximation to provide said second approximation by approximating values at each pixel of said processed image by, for each given pixel in said processed image, and each given filter, interpolating over the values corresponding to said given filter from said first approximation evaluated at those other pixels for which said vector of values of said first approximation at said other pixels are close to said vector of values at said given pixel.
13. The method of claim 12 , wherein the step of receiving comprises the step of receiving said image that has been filtered by an array of spectral filters in a mosaic pattern by a sensor array.
14. The method of claim 12 , wherein the step of processing comprises the step of processing said output of said sensor array to detect and track explosives.
15. The method of claim 12 , wherein the step of processing comprises the step of processing said output of said sensor array to detect and track biohazards.
16. The method of claim 12 , wherein the step of processing comprises the step of processing said output of said sensor array to recognize target automatically.
17. The method of claim 12 , wherein the step of performing said first interpolation comprises the step of interpolating said output of sensor array bi-linearly to provide an (n×m×L) multispectral data cube I(i, j, k), where i and j are spatial indices and k is spectral index.
18. The method of claim 12 , wherein the step of performing said second interpolation comprises the steps of:
forming a matrix
M
(
i
1
,
j
1
,
i
2
,
j
2
)
=
f
(
I
⇀
(
i
1
,
j
1
)
,
I
⇀
(
i
2
,
j
2
)
,
ɛ
)
∑
i
3
=
1
…
n
,
j
3
=
1
…
m
f
(
I
⇀
(
i
1
,
j
1
)
,
I
⇀
(
i
3
,
j
3
)
,
ɛ
)
,
where Ī(i, j) denotes the vector <I(i, j, 1), I(i, j, 2), . . . , I(i, j, L)>, with i and j fixed, k ranging over all integer values from 1 to L; and
calculating a reconstructed multispectral datacube
I
~
(
i
,
j
,
k
)
=
∑
i
1
=
1
…
n
,
j
1
=
1
…
m
M
(
i
,
j
,
i
1
,
j
1
)
·
I
(
i
1
,
j
1
,
k
)
for k ranging over all integer values from 1 to L.
19. The method of claim 18 , wherein the step of performing said second interpolation comprises the steps of forming a matrix:
M
(
i
1
,
j
1
,
i
2
,
j
2
)
=
ⅇ
-
I
_
(
i
1
,
j
1
)
-
I
_
(
i
2
,
j
2
)
2
/
ɛ
∑
i
3
=
1
…
n
,
j
3
=
1
…
m
ⅇ
-
I
_
(
i
1
,
j
1
)
-
I
_
(
i
3
,
j
3
)
2
/
ɛ
,
where Ī(i, j) denotes the vector <I(i, j, 1), I(i, j, 2), . . . , I(i, j, L)>, with i and j fixed, k ranging over all integer values from 1 to L.
20. The method of claim 19 , wherein the step of performing said second interpolation comprises the step of calculating a reconstructed multispectral datacube:
I
~
~
(
i
,
j
,
k
)
=
∑
i
1
=
i
-
C
…
i
+
C
.
j
1
=
j
-
C
…
j
+
C
M
(
i
,
j
,
i
1
,
j
1
)
·
I
(
i
1
,
j
1
,
k
)
for a fixed square region centered at (i, j) and having a length of 2C+1.