Methods and systems for camera characterization in terms of response function, color, and vignetting under non-uniform illumination
Methods and systems for describing a camera radiometrically, in terms of camera response function and vignetting, and in terms of color, suitable for non-uniform illumination set-ups. It estimates the camera response function and the camera color mapping from a single image of a generic scene with two albedos. With a second same-pose image with a different intensity of the near-light the vignetting is also estimated. The camera response function calibration involves the segmentation of the two albedos, the definition of the system of equations based on the assumption that can be made about the image formation process, and the actual estimation. For the modelling the vignetting there are three steps: computing the albedo-normalized irradiance, finding points of equal vignetting, when needed, and estimation.
1. A method for estimating a color model of an imaging device, comprised of the response function and the color mapping, under generic illumination, the method comprising:
i. acquiring at least one calibration image of a scene with at least two regions of constant albedo from an arbitrary pose;
ii. segmenting the calibration image into regions of constant albedo and grouping the pixels in each region into a set , the set of points with albedo ρ n ;
iii. deriving equations for the estimation using
f −1 ( d ( x ))= u ( x )ρ′( x )
where x are the pixel coordinates that correspond to a scene point, d(x) is the acquired image and is defined as a vector of N c elements d c (x) that correspond to values on pixel x and channel c, f −1 (d) is the inverse camera response function, a function vector of elements f c −1 (d c ), u(x) is the albedo-normalized irradiance, a scalar function, and ρ′ is the albedo vector of size N c in the color space of the camera and is defined by elements ρ′ c for each channel c such that
ρ=Mρ′
with ρ being the albedo vector of size N s in the output color space and is defined by elements ρ s for each channel s, and M being a color mapping matrix of size N s by N c and composed of rows m s T ; and
iv. determining the color model of the imaging device by solving the equations derived in step iii using a suitable solver or optimization scheme.
2. The method of claim 1 , wherein the scene used in step i is a known planar checkerboard pattern.
3. The method of claim 1 , wherein N c =1 for single-channel imaging devices, N c =3 or trichromatic imaging devices, N c >3 for multispectral and hyperspectral imaging devices, and N s =N c for an output image with the same number of channels as the input image.
4. The method of claim 1 , wherein the equations derived in step iii are defined directly from the image formation model as
f c −1 ( d c ( x ))= u ( x )ρ′ c ( x )
or, equivalently, in the logarithmic space using
log f c −1 ( d c ( x ))=log u ( x )+log ρ′ c ( x )
where log stands for a logarithm of any base and where d(x) is known and f −1 , u(x), and ρ′(x) are optimized for in step iv.
5. The method of claim 4 , wherein the scene contains at least N c albedos of known values on the output color space, which enables the estimation of M after the estimation of f −1 ,u(x), and ρ′(x) using equations of the form
ρ s =m s T ρ′
or together with the estimation of f −1 and u(x) using equations of the form
m s T f −1 ( d ( x ))= u ( x )ρ s ( x ).
6. The method of claim 1 , wherein the equations derived in step iii are defined on the isocurves of u(x), in which case the d(x) is known and f −1 is optimized for in step iv.
7. The method of claim 6 , wherein to estimate the isocurves of u(x), a parametric model h(x) is fitted to the regions of one albedo or more than one independently or using, for the purpose of incorporating more than one albedo on the same model and estimate only one set of isocurves, the following equations:
h
(
x
j
)
=
d
c
(
x
j
)
h
(
x
k
)
=
g
c
,
c
′
,
n
,
n
′
(
d
c
′
(
x
k
)
)
,
j
∈
𝒜
ρ
n
,
k
∈
𝒜
ρ
n
′
for a pair of albedos ρ n and ρ n′ , where g c,c′,n,n′ is a function that maps the values of albedo ρ c′,n′ to the model of albedo ρ′ c,n, h(x), and where only d(x) is known.
8. The method of claim 6 , wherein the isocurves are used to establish equations of the form
ρ′ c′,n′ f c −1 ( d c ( x j ))=ρ′ c,n f c′ −1 ( d c′ ( x k )), j∈ i ∩ k∈ i ∩
for a pair of albedos ρ n and ρ n′ , where i is the set of pixels crossed by isocurve i, and where d(x) is known and f −1 and ρ′(x) are optimized for in step iv.
9. The method of claim 8 , wherein the scene contains at least N c albedos of known values on the output color space, which enables the estimation of M after the estimation of f −1 and ρ′(x) using equations of the form
ρs=m s T ρ′
or together with the estimation of f −1 using equations of the form
ρ s′,n′ m s T f −1 ( d ( x j ))=ρ s,n m s′ T f −1 ( d ( x k )), j∈ i ∩ ,k∈ i ∩ .
10. The method of claim 1 , wherein the equation derivation in step iii uses a particular case matrix for the color mapping matrix comprising a diagonal matrix or a rotation matrix.
11. The method of claim 1 , wherein the solution of step iv is found using contributions from multiple calibration images, in which case steps i to iii are executed for each one of these images.
12. The method of claim 1 , wherein the determination of the color model in step iv is carried together with one or more of monotonicity constraints on the camera response function and regularization of search parameters.
13. The method of claim 1 , wherein for the purpose of having a camera response function defined for the whole range of possible pixel values d, a parametric function, an empiric model of response, or an interpolation/extrapolation method is used together with or as post processing to solving the system of equations in step iv.
14. The method of claim 1 , wherein the optimization in step iv is executed with nonlinear optimization algorithms, comprising interior point, gradient descent, and metaheuristics, for the cases of systems of equations that are nonlinear, and it is executed with convex optimization algorithms, comprising quadratic programming, for the cases of system of equations that are linear.
15. The method of claim 1 , wherein the color model of the imaging device is used to normalize the response function, by computing a new image where the pixel values d′(x) are given by
d′=f −1 ( d )
for image points x or, in the case of the color mapping matrix being also known, are given by
d′=Mf 1 ( d ).
16. The method of claim 15 , wherein the new image is computed for improved visualization and/or for further processing by other methods or algorithms, comprising shape-from-shading, photometric vision algorithms, computer-aided surgery methods, and transformations of pixel values for enhancing image properties or features, such as contrast or particular colors.
17. The method of claim 1 , wherein the output color space is a standard color space, comprising CIEXYZ and sRGB, or other predetermined color space, comprising the color space of another imaging device.
18. The method of claim 1 , wherein the algorithms presented may be used as a starting point of, integrated in, or complement other models of imaging devices, comprising multi-dimensional look-up tables.
19. A method for estimating a vignetting model of an imaging device comprising:
i. acquiring at least one set of at least two calibration images with the same relative pose and a different intensity of a near-light source;
ii. segmenting the calibration images into regions of constant albedo;
iii. computing the albedo-normalized irradiance for each pair of same-pose images, u 1 (x) and u 2 (x);
iv. deriving the equations of the vignetting model using
m ( x )= K 1 u 1 ( x )+ K 2 u 2 ( x )
where K 1 and K 2 are the two model unknowns; and
v. finding a solution for the vignetting model with the equations created in step iv.
20. The method of claim 19 , wherein, for the purpose of having additional equations, sets of points with equal vignetting are found in step iv by searching for points where one or more of both u 1 (x)/u 2 (x) and u 1 (x) are constant and both u 1 (x)/u 2 (x) and u 2 (x) are constant, which may be performed either by exhaustive search or by intersecting the isocurves of one or more of u 1 (x)/u 2 (x) and u 1 (x) and u 1 (x)/u 2 (x) and u 2 (x).
21. The method of claim 19 , wherein the vignetting estimation equations in step iv, when two or more pairs of same-pose images are available can be defined as
m ( x )= k 1 u 1 ( x )+ K 2 u 2 ( x )
m ( x )= K′ 1 u′ 1 ( x )+ k′ 2 u′ 2 ( x )
for every two pairs, which can be rewritten as
k 1 u 1 ( x )+ K 2 u 2 ( x )= K′ 1 u′ 1 ( x )+ k′ 2 u 2 ( x )
and, in this way, creating a system of equations from which the vignetting model unknowns (K 1 , K 2 , K′ 1 , and k′ 2 ) can be estimated.
22. The method of claim 19 , wherein the vignetting model of the imaging device is used to normalize any acquired images and videos, for improved visualization and/or for further processing by other methods.