Face relighting for normalization of directional lighting
View Patent ↗A method determines a direction of a principal light source in an image. An input image I is acquired of an object illuminated by a principal light source from an unknown direction. The input image includes an array of pixels, and each pixel has an intensity I(x, y). An intensity vector {right arrow over (I)} is constructed from the intensities of the input image. An albedo map ρ is defined for the object. An albedo vector {right arrow over (ρ)} is constructed from the albedo map. A shape matrix N is generated for the object. The albedo vector {right arrow over (ρ)} is multiplied by the shape matrix N to obtain a shape-albedo matrix A. Then, a direction s* to the principal light source is estimated from the intensity vector {right arrow over (I)}, the albedo vector {right arrow over (ρ)} and the shape-albedo matrix A according to an optimization s * = arg min s I → - ρ → - As .
1. A method for determining a direction of a principal light source in an image, comprising:
acquiring, with a camera, an input image I of an object illuminated by a principal light source from an unknown direction s, the input image including an array of pixels, and each pixel having an intensity I(x, y);
constructing an intensity vector {right arrow over (I)} from the intensities of the input image;
defining an albedo map ρ for the object;
constructing an albedo vector {right arrow over (ρ)} from the albedo map;
generating a shape matrix N for the object;
multiplying the albedo vector {right arrow over (ρ)} by the shape matrix N to obtain a shape-albedo matrix A;
determining an estimated direction s* to the principal light source from the intensity vector {right arrow over (I)}, the albedo vector {right arrow over (ρ)} and the shape-albedo matrix A according to an optimization
s
*
=
arg
min
s
I
→
-
ρ
→
-
As
.
2. The method of claim 1 , in which the optimization
arg
min
s
I
→
-
ρ
→
-
As
has a solution
s *=( A T A ) −1 A T ( {right arrow over (I)}−{right arrow over (ρ)}−As ),
where T is a transpose operator.
3. The method of claim 1 , in which the object belongs to a class of objects represented by a generic shape model as represented by the shape matrix N.
4. The method of claim 1 , in which the object is a face.
5. The method of claim 1 , further comprising:
illuminating the object by unknown ambient illumination.
6. The method of claim 1 , further comprising:
stacking non-zero intensities in the input image I to construct the intensity vector.
7. The method of claim 6 , further comprising:
selecting the non-zero intensities according to a predetermined threshold value.
8. The method of claim 6 , further comprising:
selecting the non-zero intensities according to a predetermined criteria.
9. The method of claim 6 , further comprising:
selecting the non-zero intensities according to a mask.
10. The method of claim 1 , further comprising:
constructing the albedo vector {right arrow over (ρ)} from a corresponding vectorized albedo map.
11. The method of claim 1 , further comprising:
constructing the albedo vector {right arrow over (ρ)} from a diffuse texture.
12. The method of claim 3 , further comprising:
generating the shape matrix by stacking surface normals of the generic shape model.
13. The method of claim 1 , in which the multiplying corresponds to an element-wise Hadamard matrix product o:
A =({right arrow over (ρ)}1 1×3 ) o N.
14. The method of claim 1 , further comprising:
estimating directly an unknown albedo from the input image using a formula
ρ
*
=
I
-
β
α
(
n
T
s
*
+
c
)
,
where α and β jointly model an intensity gain and offset of the camera respectively, c is a relative ambient illumination component, n is a unit surface normal, and T is a transpose operator.
15. The method of claim 12 , further comprising:
assigning an average albedo to a particular pixel if the intensity of the particular pixel is zero.
16. The method of claim 1 , further comprising:
generating an illumination-invariant output image under desired illumination using a formula
I o =α o {ρ*[max( n T s o ,0)+ c o ]}+β o ,
where α o and β o are a gain and offset of an output display device, c o is an intensity of the desired ambient illumination lighting in the illumination-invariant output image, n is a surface normal, s o is a direction of the desired relative ambient lighting in the illumination-invariant output image, and T is a transpose operator.
17. The method of claim 5 , further comprising:
acquiring a set of N training images of the object; and estimating properties of the unknown ambient illumination using an optimality criteria
c
*
=
arg
min
s
∑
I
=
1
N
ρ
i
(
c
)
-
1
N
∑
i
=
1
N
ρ
i
(
c
)
2
,
where c is an intensity of the unknown ambient illumination, and ρ i (c) denotes an albedo of the i th training image.
18. The method of claim 2 , further comprising:
defining a matrix P=A(A T A) −1 ; and
applying a QR-factorization P=QR to estimate an equivalent orthogonal basis for the matrix P, where the unitary matrix Q has three orthonormal columns spanning a subspace as the matrix P, and the 3×3 upper triangular matrix R defines a quality of the estimate because R −1 is a matrix square root of an error covariance.