Coupled reconstruction of refractive and opaque surfaces
View Patent ↗Systems and techniques for reconstructing one or more surfaces of an object including one or more opaque surfaces behind one or more refractive surfaces are provided. The systems and techniques may include obtaining one or more images of the object including an opaque surface located behind a refractive surface and determining one or more refractive surface constraints using the one or more images. The one or more refractive surface constraints constrain one or more characteristics of the refractive surface. The systems and techniques may further include reconstructing an opaque surface representation or a refractive surface representation using the one or more refractive surface constraints, the opaque surface representation representing the opaque surface of the object, and the refractive surface representation representing the refractive surface of the object.
1. A computer-implemented method of generating an animation model of an object, comprising:
obtaining one or more images of the object, the object including an opaque surface located behind a refractive surface;
determining one or more reflection constraints, one or more refraction constraints, and one or more position constraints for constraining generation of a refractive surface representation;
generating the refractive surface representation by optimizing a position of the refractive surface using the one or more position constraints and by optimizing one or more surface normals of the refractive surface using the one or more reflection constraints and the one or more refraction constraints; and
generating the animation model of the object, the animation model including the reconstructed refractive surface representation, wherein the refractive surface representation represents the refractive surface in the animation model of the object.
2. The method of claim 1 , further comprising reconstructing an opaque surface representation by undoing distortion introduced by the refractive surface, the opaque surface representation representing the opaque surface of the object.
3. The method of claim 1 , wherein generating the refractive surface representation includes employing an optimization method using the one or more reflection constraints, the one or more refraction constraints, and the one or more position constraints.
4. The method of claim 1 , wherein the one or more reflection constraints are determined by extracting, from the one or more images, light reflected from the refractive surface, wherein the one or more refraction constraints are determined by extracting, from the one or more images, one or more points on the opaque surface, and wherein the one or more position constraints include a point within a transition region between the refractive surface and an additional surface of the object.
5. The method of claim 1 , wherein the one or more reflection constraints are obtained by shining one or more lights onto the refractive surface.
6. The method of claim 1 , wherein the object includes an eye, the refractive surface includes a cornea part of the eye, and the opaque surface includes an iris part of the eye.
7. The method of claim 6 , further comprising reconstructing a sclera representation using the one or more images, the sclera representation representing a sclera part of the eye.
8. The method of claim 1 , wherein the object is submerged under water and the refractive surface includes a surface of the water.
9. The method of claim 1 , wherein the object is cast under a layer of material and the refractive surface includes a surface of the material.
10. A system for generating an animation model of an object, comprising:
a memory storing a plurality of instructions; and
one or more processors configurable to:
obtain one or more images of the object, the object including an opaque surface located behind a refractive surface;
determine one or more reflection constraints, one or more refraction constraints, and one or more position constraints for constraining generation of a refractive surface representation;
generate the refractive surface representation by optimizing a position of the refractive surface using the one or more position constraints and by optimizing one or more surface normals of the refractive surface using the one or more reflection constraints and the one or more refraction constraints; and
generate the animation model of the object, the animation model including the reconstructed refractive surface representation, wherein the refractive surface representation represents the refractive surface in the animation model of the object.
11. The system of claim 10 , further comprising reconstructing an opaque surface representation by undoing distortion introduced by the refractive surface, the opaque surface representation representing the opaque surface of the object.
12. The system of claim 10 , wherein generating the refractive surface representation includes employing an optimization method using the one or more reflection constraints, the one or more refraction constraints, and the one or more position constraints.
13. The system of claim 10 , wherein the one or more reflection constraints are determined by extracting, from the one or more images, light reflected from the refractive surface, wherein the one or more refraction constraints are determined by extracting, from the one or more images, one or more points on the opaque surface, and wherein the one or more position constraints include a point within a transition region between the refractive surface and an additional surface of the object.
14. The system of claim 10 , wherein the object includes an eye, the refractive surface includes a cornea part of the eye, and the opaque surface includes an iris part of the eye.
15. A non-transitory computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising:
instructions that cause the one or more processors to obtain one or more images of the object, the object including an opaque surface located behind a refractive surface;
instructions that cause the one or more processors to determine one or more reflection constraints, one or more refraction constraints, and one or more position constraints for constraining generation of a refractive surface representation;
instructions that cause the one or more processors to generating the refractive surface representation by optimizing a position of the refractive surface using the one or more position constraints and by optimizing one or more surface normals of the refractive surface using the one or more reflection constraints and the one or more refraction constraints; and
instructions that cause the one or more processors to generate the animation model of the object, the animation model including the reconstructed refractive surface representation, wherein the refractive surface representation represents the refractive surface in the animation model of the object.
16. The non-transitory computer-readable memory of claim 15 , further comprising reconstructing an opaque surface representation by undoing distortion introduced by the refractive surface, the opaque surface representation representing the opaque surface of the object.
17. The non-transitory computer-readable memory of claim 15 , wherein generating the refractive surface representation includes employing an optimization method using the one or more reflection constraints, the one or more refraction constraints, and the one or more position constraints.
18. The non-transitory computer-readable memory of claim 15 , wherein the one or more reflection constraints are determined by extracting, from the one or more images, light reflected from the refractive surface, wherein the one or more refraction constraints are determined by extracting, from the one or more images, one or more points on the opaque surface, and wherein the one or more position constraints include a point within a transition region between the refractive surface and an additional surface of the object.
19. The non-transitory computer-readable memory of claim 15 , wherein the object includes an eye, the refractive surface includes a cornea part of the eye, and the opaque surface includes an iris part of the eye.
20. The non-transitory computer-readable memory of claim 19 , further comprising instructions that cause the one or more processors to reconstruct a sclera representation using the one or more images, the sclera representation representing a sclera part of the eye.