Method and system for super resolution
View Patent ↗A method and system for enhancing the resolution of an image sensing device, including the steps of attaching a mask to a panel of detectors in an image sensing device, generating multiple fields of view, related to one another by sub-pixel shifts, acquiring multiple images with the image sensing device from the multiple fields of view, and combining the multiple images into an enhanced image of higher pixel resolution than the pixel resolutions of the multiple images.
1. A method for enhancing the resolution of an image sensing device, comprising the steps of:
applying a periodically patterned mask to an image sensing device having a pixel sensitivity function, the device comprising a periodic array of detectors having at least one of a horizontal periodic spacing and a vertical periodic spacing,
the periodically patterned mask having at least one of a horizontal period which is an integer multiple of said detectors' horizontal periodic spacing and a vertical period which is an integer multiple of the detectors' vertical periodic spacing,
the mask modifying the pixel sensitivity function of the image sensing device, thereby to define a modified pixel sensitivity function of the device whose Fourier transform does not have zeros at frequencies that are between 0 and the maximal frequency at which no significant aliasing occurs;
using the image sensing device with the mask to acquire multiple images of a scene from corresponding multiple fields of view related to one another by sub-pixel shifts, the multiple images defining sub-pixel samples; and
combining the multiple images into an enhanced image of higher pixel resolution than the pixel resolutions of the multiple images, including:
computing a Fourier transform of said samples;
dividing said Fourier transform of said samples by a Fourier transform of said modified pixel sensitivity function, thereby to define a quotient; and
computing an inverse Fourier transform of the quotient, thereby to define the enhanced image.
2. The method of claim 1 wherein the image sensing device is a scanner.
3. The method of claim 2 wherein the image sensing device is a barcode reader.
4. The method of claim 1 wherein the image sensing device is a CCD camera.
5. The method of claim 1 wherein the mask is a fine transmission grating.
6. The method of claim 1 wherein said combining step uses a Gabor transform.
7. The method of claim 1 wherein said combining step uses a wavelet transform.
8. The method of claim 1 wherein said combining step uses a Mellin transform.
9. A method according to claim 1 , comprising the steps of:
creating replicas of fields of view using an optical element attached to the image sensing device;
acquiring multiple images with the sensing device from the replicas of fields of view; and
combining the multiple images acquired from the replicas into an enhanced image of higher pixel resolution than the pixel resolutions of the multiple images acquired from the replicas.
10. The method of claim 9 wherein the optical element is a diffractive optical element.
11. The method of claim 9 wherein the optical element is a reflective optical element.
12. The method of claim 9 wherein the optical element is a combined diffractive and reflective optical element.
13. The method of claim 9 wherein the optical element produces an effect of a grating.
14. The method of claim 9 wherein the optical element is multi-faceted.
15. The method of claim 9 wherein the replicas of fields of views are non-overlapping.
16. A method according to claim 1 wherein said applying of a mask comprises bringing the mask into a substantially touching engagement with a detector plane defined by the periodic array of detectors.
17. A method for enhancing the resolution of an image sensing device, comprising the steps of:
attaching a mask to a panel of detectors in an image sensing device, the mask being a fine transmission grating, the panel of detectors comprising a periodic array of detectors, and the mask being periodic with period equal to that of the periodic array of detectors;
generating multiple fields of view, the multiple fields of view being related to one another by sub-pixel shifts;
acquiring multiple images with the image sensing device from the multiple fields of view; and
combining the multiple images into an enhanced image of higher pixel resolution than the pixel resolutions of the multiple images.
18. A method for enhancing the resolution of an image sensing device, comprising the steps of:
attaching a mask to a panel of detectors in an image sensing device, the mask being a fine transmission grating, the panel of detectors comprising a periodic array of detectors, and the mask being periodic with period twice that of the periodic array of detectors;
generating multiple fields of view, the multiple fields of view being related to one another by sub-pixel shifts;
acquiring multiple images with the image sensing device from the multiple fields of view; and
combining the multiple images into an enhanced image of higher pixel resolution than the pixel resolutions of the multiple images.
19. A method for enhancing the resolution of an image sensing device, comprising the steps of:
attaching a periodic to a panel of detectors in an image sensing device, the mask being a fine transmission grating;
generating multiple fields of view, the multiple fields of view being related to one another by sub-pixel shifts;
acquiring multiple images with the image sensing device from the multiple fields of view;
combining the multiple images into an enhanced image of higher pixel resolution than the pixel resolutions of the multiple images, and
determining the number of slits, the positions of the slits and the widths of the slits in the fine transmission grating.
20. The method of claim 19 wherein said determining step determines the number of slits, the positions of the slits and the widths of the slits, so that the mask eliminates zeroes in a Fourier transform of a function representing a sensitivity of a pixel, within a prescribed range of frequencies.
21. The method of claim 20 wherein the widths of the slits are constrained so as not to block more than a prescribed fraction of light energy from reaching the detectors.
22. The method of claim 20 wherein the widths of the slits are constrained so as not to be smaller than a prescribed minimum width.
23. The method of claim 20 wherein said determining step uses a weight function specifying relative weights to be placed on values for the widths of the slits.
24. The method of claim 20 wherein the mask is of the form
m
(
x
)
=
1
-
∑
i
=
1
M
rect
(
x
δ
x
i
)
⊗
δ
(
x
-
x
i
)
.
25. A method for enhancing the resolution of an image sensing device, comprising the steps of:
attaching a periodic to a panel of detectors in an image sensing device;
generating multiple fields of view, the multiple fields of view being related to one another by sub-pixel shifts;
acquiring multiple images with the image sensing device from the multiple fields of view; and
combining the multiple images into an enhanced image of higher pixel resolution than the pixel resolutions of the multiple images,
wherein said generating step uses natural vibrations of a platform supporting the sensing device.
26. A method for enhancing the resolution of an image sensing device, comprising the steps of:
attaching a periodic to a panel of detectors in an image sensing device;
generating multiple fields of view, the multiple fields of view being related to one another by sub-pixel shifts;
acquiring multiple images with the image sensing device from the multiple fields of view; and
combining the multiple images into an enhanced image of higher pixel resolution than the pixel resolutions of the multiple images,
wherein said generating step uses at least one mirror to shift fields of view of the image sensing device by sub-pixel shifts.
27. A method for enhancing the resolution of an image sensing device, comprising the steps of:
attaching a periodic to a panel of detectors in an image sensing device:
generating multiple fields of view, the multiple fields of view being related to one another by sub-pixel shifts;
acquiring multiple images with the image sensing device from the multiple fields of view; and
combining the multiple images into an enhanced image of higher pixel resolution than the pixel resolutions of the multiple images,
wherein said combining step is applied separately to rows and columns of pixel data.
28. The method of claim 27 wherein said combining step uses the approximation
U
(
ⅇ
jω
)
≈
Y
(
ⅇ
jω
)
G
(
-
j
ω
K
Δ
x
)
,
to determine the rows and columns of pixel data of the enhanced image.
29. The method of claim 28 wherein said combining step comprises applying an inverse Fourier transform to values of U(e jω ).
30. The method of claim 27 wherein said combining step uses the approximation
U
(
ⅇ
jω
)
≈
1
2
[
Y
(
ⅇ
j
ω
2
)
/
G
(
-
j
ω
K
Δ
x
)
+
Y
(
ⅇ
j
ω
-
2
π
2
)
/
G
(
-
j
(
ω
-
2
π
)
K
Δ
x
)
]
,
0
<
ω
<
π
,
for positive values of ω, and
U
(
ⅇ
jω
)
≈
1
2
[
Y
(
ⅇ
j
ω
2
)
/
G
(
-
j
ω
K
Δ
x
)
+
Y
(
ⅇ
j
ω
-
2
π
2
)
/
G
(
-
j
(
ω
-
2
π
)
K
Δ
x
)
]
-
π
<
ω
<
0
,
for negative values of ω, to determine the rows and columns of pixel data of the enhanced image.
31. The method of claim 30 wherein said combining step comprises applying an inverse Fourier transform to values of U(e jω ).
32. The method of claim 27 wherein said combining step uses the approximation
U
(
ⅇ
jω
)
≈
[
Y
2
(
ⅇ
jω
)
δ
2
π
k
Δ
x
G
1
(
-
j
ω
K
Δ
x
)
-
Y
1
(
ⅇ
jω
)
δ
2
π
k
Δ
x
G
2
(
-
j
ω
K
Δ
x
)
]
/
[
G
2
(
-
j
ω
K
Δ
x
)
δ
2
π
k
Δ
x
G
1
(
-
j
ω
K
Δ
x
)
-
G
1
(
-
j
ω
K
Δ
x
)
δ
2
π
k
Δ
x
G
2
(
-
j
ω
K
Δ
x
)
]
for positive values of ω, 0<ω<π, and
U
(
ⅇ
j
ω
)
≈
[
Y
2
(
ⅇ
j
ω
)
δ
2
π
K
Δ
x
G
1
(
-
j
ω
K
Δ
x
)
-
Y
1
(
ⅇ
j
ω
)
δ
-
2
π
K
Δ
x
G
2
(
-
j
ω
K
Δ
x
)
]
/
[
G
2
(
-
j
ω
K
Δ
x
)
δ
-
2
π
K
Δ
x
G
1
(
-
j
ω
K
Δ
x
)
δ
-
2
π
K
Δ
x
G
2
(
-
j
ω
K
Δ
x
)
]
for negative values of ω, −π<Ω<0, and wherein the terms δ Ψ G(−jΩ) denote the difference
δ Ψ G(−jΩ)=G(−jΩ)−G(−j(Ω−Ψ),
to determine the rows and columns of pixel data of the enhanced image.
33. The method of claim 32 wherein said combining step comprises applying an inverse Fourier transform to values of U(e jω ).
34. A system for enhancing the resolution of an image sensing device, comprising:
an image sensing device comprising a periodic array of detectors having at least one of a horizontal periodic spacing and a vertical periodic spacing, said image sensing device having a pixel sensitivity function;
a periodically patterned mask attached to said image sensing device, said mask having at least one of a horizontal period which is an integer multiple of said detectors' horizontal periodic spacing and a vertical period which is an integer multiple of the detectors' vertical periodic spacing, said mask being operative to modify said pixel sensitivity function of said image sensing device, thereby to define a modified pixel sensitivity function of said device whose Fourier transform does not have zeros at frequencies that are between 0 and the maximal frequency at which no significant aliasing occurs;
image acquisition circuitry, operative to acquire multiple images of a scene from corresponding multiple fields of view related to one another by sub-pixel shifts, said multiple images defining sub-pixel samples; and
a combiner combining the multiple images into an enhanced image of higher pixel resolution than the pixel resolutions of the multiple images, said combiner being operative
to compute a Fourier transform of said samples;
to divide said Fourier transform of said samples by a Fourier transform of said modified pixel sensitivity function, thereby to define a quotient; and
to compute an inverse Fourier transform of the quotient, thereby to define the enhanced image.
35. The system of claim 34 wherein said image sensing device is a scanner.
36. The system of claim 35 wherein said image sensing device is a barcode reader.
37. The system of claim 34 wherein said image sensing device is a CCD camera.
38. The system of claim 32 wherein said mask is a fine transmission grating.
39. The system of claim 34 wherein said combiner uses a Gabor transform.
40. The system of claim 34 wherein said combiner uses a wavelet transform.
41. The system of claim 32 wherein said combiner uses a Mellin transform.
42. A system according to claim 34 wherein said multiple images are acquired by creating replicas of fields of view using an optical element attached to the image sensing device.
43. The system of claim 42 wherein said optical element is a diffractive optical element.
44. The system of claim 42 wherein said optical element is a reflective optical element.
45. The system of claim 42 wherein said optical element is a combined diffractive and reflective optical element.
46. The system of claim 42 wherein said optical element produces an effect of a grating.
47. The system of claim 42 wherein said optical element is multi-faceted.
48. The system of claim 42 wherein the replicas of fields of view are non-overlapping.
49. A system for enhancing the resolution of an image sensing device, comprising:
an image sensing device comprising a panel of detectors said panel of detectors comprising a periodic array of detectors;
a mask attached to said panel of detectors said mask being a fine transmission grating, and said mask being periodic with period equal to that of said periodic array of detectors;
a motion generator generating multiple fields of view the multiple fields of view being related to one another by sub-pixel shifts;
image acquisition circuitry housed within said image sensing device acquiring multiple images from the multiple fields of view; and
a combiner combining the multiple images into an enhanced image of higher pixel resolution than the pixel resolutions of the multiple images.
50. A system for enhancing the resolution of an image sensing device, comprising:
an image sensing device comprising a panel of detectors, said panel of detectors comprising a periodic array of detectors;
a mask attached to said panel of detectors, said mask being a fine transmission grating, and said mask being periodic with period twice that of said periodic array of detectors;
a motion generator generating multiple fields of view, the multiple fields of view being related to one another by sub-pixel shifts;
image acquisition circuitry housed within said image sensing device acquiring multiple images from the multiple fields of view; and
a combiner combining the multiple images into an enhanced image of higher pixel resolution than the pixel resolutions of the multiple images.
51. A system for enhancing the resolution of an image sensing device, comprising:
an image sensing device comprising a panel of detectors;
a periodic attached to said panel of detectors, said mask being a fine transmission grating;
a motion generator generating multiple fields of view, the multiple fields of view being related to one another by sub-pixel shifts;
image acquisition circuitry housed within said image sensing device acquiring multiple images from the multiple fields of view;
a combiner combining the multiple images into an enhanced image of higher pixel resolution than the pixel resolutions of the multiple images; and
a grating generator determining the number of slits, the positions of the slits and the widths of the slits in the fine transmission grating.
52. The system of claim 51 wherein said grating generator determines the number of slits, the positions of the slits and the widths of the slits, so that said mask eliminates zeroes in a Fourier transform of a function representing a sensitivity of a pixel.
53. The system of claim 52 wherein the widths of the slits are constrained so as not to block more than a prescribed fraction of light energy from reaching the detectors.
54. The system of claim 52 wherein the widths of the slits are constrained so as not to be smaller than a prescribed minimum width.
55. The system of claim 52 wherein said grating generator uses a weight function specifying relative weights to be placed on values for the widths of the slits.
56. The system of claim 52 wherein said mask is of the form
m
(
x
)
=
1
-
∑
i
=
1
M
rect
(
x
δ
x
i
)
⊗
δ
(
x
-
x
i
)
.
57. A system for enhancing the resolution of an image sensing device, comprising:
an image sensing device comprising a panel of detectors;
a periodic attached to said panel of detectors;
a motion generator generating multiple fields of view, the multiple fields of view being related to one another by sub-pixel shifts, said motion generator using natural vibrations of a platform supporting the sensing device;
image acquisition circuitry housed within said image sensing device acquiring multiple images from the multiple fields of view; and
a combiner combining the multiple images into an enhanced image of higher pixel resolution than the pixel resolutions of the multiple images.
58. A system for enhancing the resolution of an image sensing device, comprising:
an image sensing device comprising a panel of detectors;
a periodic attached to said panel of detectors;
a motion generator generating multiple fields of view, the multiple fields of view being related to one another by sub-pixel shifts, said motion generator using at least one mirror to shift fields of view of the image sensing device by sub-pixel shifts;
image acquisition circuitry housed within said image sensing device acquiring multiple images from the multiple fields of view; and
a combiner combining the multiple images into an enhanced image of higher pixel resolution than the pixel resolutions of the multiple images.
59. A system for enhancing the resolution of an image sensing device, comprising:
an image sensing device comprising a panel of detectors:
a periodic attached to said panel of detectors:
a motion generator generating multiple fields of view, the multiple fields of view being related to one another by sub-pixel shifts;
image acquisition circuitry housed within said image sensing device acquiring multiple images from the multiple fields of view; and
a combiner combining the multiple images into an enhanced image of higher pixel resolution than the pixel resolutions of the multiple images, said combiner being applied separately to rows and columns of pixel data.
60. The system of claim 59 wherein said combiner uses the approximation
U
(
ⅇ
j
ω
)
≈
Y
(
ⅇ
j
ω
)
G
(
-
j
ω
K
Δ
x
)
,
to determine the rows and columns of pixel data of the enhanced image.
61. The system of claim 60 wherein said combiner applies an inverse Fourier transform to values of U(e jω ).
62. The system of claim 59 wherein said combiner uses the approximation
U
(
ⅇ
j
ω
)
≈
1
2
[
Y
(
ⅇ
j
ω
2
)
/
G
(
-
j
ω
K
Δ
x
)
+
Y
(
ⅇ
j
ω
-
2
π
2
)
/
G
(
-
j
(
ω
-
2
π
)
K
Δ
x
)
]
,
0
<
ω
<
π
,
for positive values of ω, and
U
(
ⅇ
j
ω
)
≈
1
2
[
Y
(
ⅇ
j
ω
2
)
/
G
(
-
j
ω
K
Δ
x
)
+
Y
(
ⅇ
j
ω
+
2
π
2
)
/
G
(
-
j
(
ω
+
2
π
)
K
Δ
x
)
]
-
π
<
ω
<
0
,
for negative values of ω, to determine the rows and columns of pixel data of the enhanced image.
63. The system of claim 62 wherein said combiner comprises applying an inverse Fourier transform to values of U(e jω ).
64. The system of claim 59 wherein said combiner uses the approximation
U
(
ⅇ
j
ω
)
≈
[
Y
2
(
ⅇ
j
ω
)
δ
2
π
K
Δ
x
G
1
(
-
j
ω
K
Δ
x
)
-
Y
1
(
ⅇ
j
ω
)
δ
-
2
π
k
Δ
x
G
2
(
-
j
ω
K
Δ
x
)
]
/
[
G
2
(
-
j
ω
K
Δ
x
)
δ
2
π
K
Δ
x
G
1
(
-
j
ω
K
Δ
x
)
-
G
1
(
-
j
ω
K
Δ
x
)
δ
2
π
K
Δ
x
G
2
(
-
j
ω
K
Δ
x
)
]
for positive values of ω, 0<ω<π, and
U
(
ⅇ
j
ω
)
≈
[
Y
2
(
ⅇ
j
ω
)
δ
-
2
π
K
Δ
x
G
1
(
-
j
ω
K
Δ
x
)
-
Y
1
(
ⅇ
j
ω
)
δ
-
2
π
k
Δ
x
G
2
(
-
j
ω
K
Δ
x
)
]
/
[
G
2
(
-
j
ω
K
Δ
x
)
δ
-
2
π
K
Δ
x
G
1
(
-
j
ω
K
Δ
x
)
-
G
1
(
-
j
ω
K
Δ
x
)
δ
-
2
π
K
Δ
x
G
2
(
-
j
ω
K
Δ
x
)
]
for negative values of ω, −π<ω<0, and wherein the terms δ Ψ (−jΩ) denote the difference
δ Ψ G(−jΩ)=G(−jΩ)−G(−j(Ω−Ψ),
to determine the rows and columns of pixel data of the enhanced image.
65. The system of claim 64 wherein said combiner comprises applying an inverse Fourier transform to values of U(e jω ).