Systems and methods for three dimensional geometric reconstruction of captured image data
View Patent ↗In various embodiments, methods, systems, and computer program products for processing digital images captured by a mobile device are disclosed. Myriad features enable and/or facilitate processing of such digital images using a mobile device that would otherwise be technically impossible or impractical, and furthermore address unique challenges presented by images captured using a camera rather than a traditional flat-bed scanner, paper-feed scanner, or multifunction peripheral. Notably, the presently disclosed systems and techniques enable three-dimensional reconstruction of objects depicted in image captured using a camera of a mobile device. The reconstruction corrects or compensates for perspective distortion caused by camera-based capture.
1. A method of reconstructing a digital image, the method comprising:
receiving the digital image comprising a digital representation of an object bounded by a tetragon;
dividing the digital image into a plurality of digital image segments;
performing a segmentation analysis on the plurality of digital image segments;
correcting curvature in the tetragon to form a quadrilateral based at least in part on the segmentation analysis;
correcting projective effects in the quadrilateral to form a rectangle based at least in part on the segmentation analysis; and
estimating an absolute size of the object based on intrinsic parameters of a capture device used to capture the digital image.
2. The method as recited in claim 1 , wherein correcting the curvature comprises transforming one or more portions of the tetragon based at least in part on intrinsic tetragon coordinates using a four-point algorithm.
3. The method as recited in claim 1 , wherein the digital image is characterized by a capture angle of about 30 degrees or more deviation from normal with respect to the object.
4. The method as recited in claim 1 , wherein correcting the projective effects comprises one or more of singular value decomposition (SVD) and direct linear transformation (DLT).
5. The method as recited in claim 1 , further comprising estimating a two-dimensional pixel position for each of up to four pixels in the digital image, each of the up to four pixels corresponding to one of four corners of one or more of the object and the tetragon.
6. The method as recited in claim 5 , wherein estimating the two-dimensions pixel position for each of the four pixels is based on a predetermined correspondence between each of the four pixels and four corresponding pixels of a reference image,
wherein the reference image depicts the digital representation of the object from a perspective corresponding to a capture angle normal to the object, and
wherein each of the four corresponding pixels of the reference image corresponds to one of four corners of the digital representation of the object in the reference image.
7. The method as recited in claim 1 , further comprising estimating a two-dimensional pixel position for each of four corners of one or more of the object and the tetragon, and
wherein at least one of the four corners is characterized by being one or more of:
obstructed in the digital image;
not depicted in the digital image; and
blurred beyond recognition in the digital image.
8. The method as recited in claim 1 ,
wherein correcting the projective effects comprises estimating a homography transform H,
wherein estimating H comprises detecting one or more point correspondences p i P i ′ with p i =(x i ,y i ,1) T ,
wherein estimating H comprises detecting four point correspondences, each point correspondence p i P i ′ corresponding to a corner position p i of the tetragon and a respective corner position P′ i of the rectangle, and
wherein H satisfies the expression λp i =HP′ i , where
λ=a focal distance between a three-dimensional reference coordinate position (X, Y, Z) and a corresponding reference coordinate position of a capture device;
p i =(x i ,y i ,1) T ;
P′ I =(X i ,Y i ,1) T , and
i is the respective index feature of the point p or P′ in the corresponding image.
9. The method as recited in claim 1 , wherein correcting the projective effects comprises performing a non-linear refinement.
10. The method as recited in claim 1 , further comprising outputting the reconstructed digital image, wherein the reconstructed digital image is characterized by a pixel location error of about 5 pixels or less relative to a scanned 500-DPI image of the object captured using a capture angle normal to the object.
11. The method as recited in claim 1 , wherein at least one side of the tetragon is characterized by a second degree or higher order polynomial.
12. The method as recited in claim 1 , wherein correcting the projective effects comprises normalizing coordinates of point-correspondences between the digital image and a reconstructed image.
13. The method as recited in claim 1 , wherein the object is a document characterized by a known height-to-width ratio.
14. The method as recited in claim 1 , wherein the estimating employs an intrinsic parameter matrix A representative of the intrinsic capture device parameters.
15. The method as recited in claim 14 , wherein the intrinsic parameter matrix A is characterized by an expression
A
=
(
a
c
d
b
e
1
)
,
and
wherein a and b are each scaling factors which comprise focal length information corresponding to the capture device.
16. The method as recited in claim 15 , wherein a=f/dx
wherein b=f/dy,
wherein f is the focal length,
wherein dx is a horizontal scaling factor,
wherein dy is a vertical scaling factor,
wherein c is a skew parameter, and
wherein (d, e) are the coordinates of a principal point in the digital image.
17. The method as recited in claim 15 , wherein c =0, wherein either:
(d, e) are the coordinates of a principal point in the digital image, or
d=0 and e=0.
18. A system, comprising:
a processor configured to execute logic; and
logic configured to receive a digital image comprising a digital representation of an object bounded by a tetragon;
logic configured to correct curvature in the tetragon to form a quadrilateral; and
logic configured to correct projective effects in the quadrilateral to form a rectangle,
wherein correcting the projective effects is based at least in part on estimating corner positions of the object; and
logic configured to estimate an absolute size of the object based on intrinsic parameters of a capture device used to capture the digital image, and
wherein at least one corner of the object for which the corner position is estimated is characterized by being one or more of:
obstructed in the digital image;
not depicted in the digital image; and
blurred beyond recognition in the digital image.
19. A computer program product comprising a non-transitory computer readable storage medium having computer readable program code stored thereon, the computer readable program code comprising:
computer readable program code configured to receive a digital image comprising a digital representation of an object bounded by a tetragon;
computer readable program code configured to correct curvature in the tetragon to form a quadrilateral;
computer readable program code configured to correct projective effects in the quadrilateral to form a rectangle; and
computer readable program code configured to estimate an absolute size of the object based on intrinsic parameters of a capture device used to capture the digital image.