Surface normal estimation for use in rendering an image

1. A method of determining surface normal estimates for a surface of an object which is visible in one or more images of a scene, wherein the object is represented by geometry constructed from the one or more images of the scene, the method comprising:

obtaining surface normal estimates for the surface of the object, the surface normal estimates representing a first surface normal field which comprises a non-conservative component and a conservative component;

constructing a height map for said surface of the object based on the conservative component of the first surface normal field;

using the height map to determine refined surface normal estimates for the surface of the object, the refined surface normal estimates representing a conservative surface normal field for said surface of the object; and

storing the refined surface normal estimates for subsequent use in rendering an image of the object.

2. The method of claim 1 wherein said obtaining surface normal estimates for the surface of the object comprises:

determining a lighting estimate for the scene;

determining surface shading estimates for the surface of the object; and

determining the surface normal estimates for the surface of the object using the determined lighting estimate and the determined surface shading estimates for the surface of the object.

3. The method of claim 2 wherein said determining the surface normal estimates also uses coarse surface normals for the object predicted from the geometry constructed from the one or more images of the scene.

4. The method of claim 3 wherein said determining the surface normal estimates comprises, for each of a plurality of sample positions, finding the surface normal which minimizes an error metric comprising a measure of the difference between the determined surface shading estimate for the sample position and the determined lighting estimate for the direction of the surface normal, wherein the error metric further comprises a regularization term which is a function of a difference between the surface normal for the sample position and a coarse surface normal for the sample position predicted from the geometry constructed from the one or more images of the scene.

5. The method of claim 1 further comprising analysing the one or more images of the scene to construct the geometry representing the object.

6. The method of claim 1 wherein the first surface normal field is a non-conservative field comprising a non-conservative component and the conservative component.

7. The method of claim 1 wherein divergence of the first surface normal field is used in said constructing a height map for the surface of the object, such that the height map is constructed based on the conservative component of the first surface normal field.

8. The method of claim 1 wherein the height map is constructed using a version of Poisson's equation, wherein an iterative relaxation method is used to determine a height at each of a plurality of sample positions based on the heights at neighbouring sample positions and based on divergence of the first surface normal field.

9. The method of claim 8 wherein, on the i th iteration of the iterative relaxation method, the height H i (x, y), at a sample position (x, y) of a texture for the surface of the object is given by:

H i ( x,y )=¼ {H i-1 ( x+δ,y )+ H i-1 ( x+δ,y )+ H i-1 ( x+δ,y )+ H 1-1 ( x+δ,y )−δ 2 ∇·N i }

where δ is an interval between adjacent sample positions and N i is the first surface normal field.

10. The method of claim 8 wherein the relaxation method is terminated after a number of iterations such that the relaxation method terminates before convergence.

11. The method of claim 1 wherein said using the height map to determine refined surface normal estimates for the surface of the object comprises determining gradients of the height map.

12. The method of claim 11 wherein the refined surface normal estimates, N r , are determined using the height map, H, according to the equation:

∇ H=−N r .

13. The method of claim 1 wherein the one or more images of the scene are captured from one or more camera viewpoints, and wherein the stored refined surface normal estimates are for subsequent use in rendering an image of the object under arbitrary lighting conditions from a rendering viewpoint.

14. The method of claim 13 wherein the refined surface normal estimates are stored with an intrinsic colour component of a relightable texture for subsequent use in rendering an image of the object under arbitrary lighting conditions from a rendering viewpoint.

15. An image processing system configured to determine surface normal estimates for a surface of an object which is visible in one or more images of a scene, wherein the image processing system is configured to obtain geometry representing the object, wherein the geometry is constructed from the one or more images of the scene, the image processing system comprising:

first surface normal logic configured to obtain surface normal estimates for the surface of the object, the surface normal estimates representing a first surface normal field which comprises a non-conservative component and a conservative component;

height map construction logic configured to construct a height map for said surface of the object based on the conservative component of the first surface normal field;

second surface normal logic configured to use the height map to determine refined surface normal estimates for the surface of the object, the refined surface normal estimates representing a conservative surface normal field for said surface of the object; and

a store configured to store the refined surface normal estimates for subsequent use in rendering an image of the object.

16. The image processing system of claim 15 wherein the image processing system further comprises:

scene lighting estimation logic configured to determine a lighting estimate for the scene; and

shading estimation logic configured to determine surface shading estimates for the surface of the object;

wherein the first surface normal logic is configured to obtain the surface normal estimates for the surface of the object by determining the surface normal estimates for the surface of the object using the determined lighting estimate and the determined surface shading estimates for the surface of the object.

17. The image processing system of claim 16 wherein the first surface normal logic is configured to also use coarse surface normals for the object predicted from the geometry constructed from the one or more images of the scene to determine the surface normal estimates.

18. The image processing system of claim 15 further comprising scene analysis logic configured to analyse the one or more images of the scene to construct the geometry representing the object.

19. The image processing system of claim 15 wherein the height map construction logic is configured to use divergence of the first surface normal field to construct the height map for the surface of the object, to thereby construct the height map based on the conservative component of the first surface normal field.

20. A non-transitory computer readable storage medium having stored thereon computer readable code that causes a processor to either:

(i) determine surface normal estimates for a surface of an object which is visible in one or more images of a scene, wherein the object is represented by geometry constructed from the one or more images of the scene, by:

obtaining surface normal estimates for the surface of the object, the surface normal estimates representing a first surface normal field which comprises a non-conservative component and a conservative component;

constructing a height map for said surface of the object based on the conservative component of the first surface normal field;

using the height map to determine refined surface normal estimates for the surface of the object, the refined surface normal estimates representing a conservative surface normal field for said surface of the object; and

storing the refined surface normal estimates for subsequent use in rendering an image of the object; or

(ii) generate a processing block configured to determine surface normal estimates for a surface of an object which is visible in one or more images of a scene, wherein the object is represented by geometry constructed from the one or more images of the scene, by:

obtaining surface normal estimates for the surface of the object, the surface normal estimates representing a first surface normal field which comprises a non-conservative component and a conservative component;

constructing a height map for said surface of the object based on the conservative component of the first surface normal field;

using the height map to determine refined surface normal estimates for the surface of the object, the refined surface normal estimates representing a conservative surface normal field for said surface of the object; and

storing the refined surface normal estimates for subsequent use in rendering an image of the object.