Systems, methods, and media for directly recovering planar surfaces in a scene using structured light
In accordance with some embodiments, systems, methods and media for directly recovering planar surfaces in a scene using structured light are provided. In some embodiments, a system comprises: a light source; an image sensor; a processor programmed to: cause the light source to emit a pattern comprising a pattern feature with two line segments that intersect on an epipolar line; cause the image sensor to capture an image including the pattern; identify an image feature in the image, the image feature comprising two intersecting line segments that intersect at a point in the image that corresponds to the first epipolar line; estimate a plane hypothesis associated with the pattern feature based on properties of the pattern feature and properties of the image feature, the plane hypothesis associated with a set of parameters characterizing a plane; and identify a planar surface in the scene based on the plane hypothesis.
1. A system for recovering planes in a scene, the system comprising:
a light source;
an image sensor comprising an array of pixels;
at least one hardware processor that is programmed to:
cause the light source to emit a two-dimensional light pattern toward the scene,
wherein the two-dimensional light pattern comprises a pattern feature that is disposed on a first epipolar line, the pattern feature comprising two intersecting line segments that intersect at a point that is located on the first epipolar line, and
wherein a first line segment of the pattern feature is angled in a range of 35° to 55° with respect to the first epipolar line, and the second line segment of the pattern feature is angled in a range of 125° to 145° with respect to the first epipolar line;
cause the image sensor to capture an image of the scene including at least a portion of the light pattern;
identify an image feature in the image, the image feature comprising two intersecting line segments that intersect at a point in the image that corresponds to the first epipolar line;
identify at least the pattern feature as potentially corresponding to the image feature based on the image feature and the pattern feature both being associated with the first epipolar line;
estimate a plane hypothesis associated with the pattern feature based on properties of the pattern feature and properties of the image feature, wherein the plane hypothesis is associated with a set of parameters that characterize a plane; and
identify a planar surface in the scene based on the plane hypothesis.
2. The system of claim 1 ,
wherein the two-dimensional light pattern comprises a set of N pattern features {P 1 , . . . , P N }, including the pattern feature, that are each disposed on the first epipolar line; and
wherein the at least one hardware processor is further programmed to:
identify the set of N pattern features as potentially corresponding to the image feature based on the image feature and the set of N pattern features all being associated with the first epipolar line;
estimate a first set of N plane hypotheses each associated with a respective pattern feature of the set of N pattern features, each plane hypothesis of the first set of N plane hypotheses based on properties of the image feature and properties of the respective pattern feature,
wherein each plane hypothesis of the first set of N plane hypotheses is associated with a set of parameters that characterize a plane;
identify a second image feature in the image that is associated with the first epipolar line, the second image feature comprising two intersecting line segments that intersect at a point that corresponds to the first epipolar line;
identify the set of N pattern features as potentially corresponding to the second image feature based on the second image feature and the set of N pattern features all being associated with the first epipolar line;
estimate a second set of N plane hypotheses each associated with a respective pattern feature of the set of N pattern features, each plane hypothesis of the second set of N plane hypotheses based on properties of the second image feature and properties of the respective pattern feature,
wherein each plane hypothesis of the second set of N plane hypotheses comprises a set of parameters that characterize a plane; and
identify the planar surface in the scene based on a plane hypothesis that is a member of both the first set of N plane hypotheses and the second set of N plane hypotheses.
3. The system of claim 2 , wherein N >2, and a distance along the first epipolar line between pattern feature P 1 and pattern feature P 2 is different than a distance along the first epipolar line between pattern feature P N−1 and pattern feature P N , such that the set of N pattern features {P 1 , . . . , P N } are non-uniformly spaced.
4. The system of claim 3 , wherein the two-dimensional light pattern comprises a second set of N pattern features { N+1 , . . . , P 2N } that are each disposed on a second epipolar line, and
wherein a distance along the second epipolar line between pattern feature P N1 and pattern feature P N+2 is different than a distance along the second epipolar line between pattern feature P 2N−1 and pattern feature P 2N , and
the distance along the second epipolar line between pattern feature P N1 and pattern feature P N+2 is different than a distance along the first epipolar line between pattern feature P 1 and pattern feature P 2 , such that the second set of N pattern features {P N+1 , . . . , P 2N } are non-uniformly spaced and are spaced differently than the set of N pattern features {P 1 , . . . ,P N }.
5. The system of claim 1 , wherein the at least one hardware processor is further programmed to determine that the image feature is associated with the first epipolar line based on a proximity between the point at which the two line segments intersect and the first epipolar line.
6. The system of claim 1 ,
wherein the light source comprises a plurality of rows, and the image sensor comprises a second plurality of rows, and
the light source and the image sensor are rectified such that each row of the plurality of rows shares an epipolar line with a row of the second plurality of rows.
7. The system of claim 1 ,
wherein the properties of the pattern feature comprise a first vector u p corresponding to a first line segment of the pattern feature, a second vector v p corresponding to a second line segment of the pattern feature, and coordinates p p of the point at which the first vector u p and the second vector v p intersect, and
wherein the properties of the image feature comprise a third vector u c corresponding to a first line segment of the image feature, a fourth vector v c corresponding to a second line segment of the image feature, and coordinates p c of the point at which the third vector u c and the fourth vector v c intersect.
8. The system of claim 7 , wherein the set of parameters associated with the plane hypothesis define a plane Π and include a distance D corresponding to a shortest distance between an origin c c of a camera coordinate system to the plane Π, a polar angle θ between a plane normal vector n and a z axis extending through c c and a center of an image plane defined by the image sensor, and an azimuthal φ between the plane normal vector n and an x axis extending along a horizontal direction of the image plane.
9. The system of claim 8 , wherein the processor is further programmed to:
determine the plane normal vector n using the relationship
n
=
(
(
p
p
×
v
p
)
×
(
p
c
×
v
c
)
)
×
(
(
p
p
×
u
p
)
×
(
p
c
×
u
c
)
)
(
(
p
p
×
v
p
)
×
(
p
c
×
v
c
)
)
×
(
(
p
p
×
u
p
)
×
(
p
c
×
u
c
)
)
;
and
D
=
b
n
T
p
p
p
p
x
-
p
c
x
-
n
T
c
p
,
determine the distance D using the relationship where b is a distance between the origin c c of the camera coordinate system and an origin c p of a pattern coordinate system.
10. The system of claim 1 , wherein the processor is further programmed to:
cause the image sensor to capture a second image of the scene that does not include the light pattern;
calculate a difference image based on a difference between the image and the second image; and
identify the image feature in the image using the difference image.
11. A method for recovering planes in a scene, the method comprising:
causing a light source to emit a two-dimensional light pattern toward the scene,
wherein the two-dimensional light pattern comprises a pattern feature that is disposed on a first epipolar line, the pattern feature comprising two intersecting line segments that intersect at a point that is located on the first epipolar line, and
wherein a first line segment of the pattern feature is angled in a range of 35° to 55° with respect to the first epipolar line, and the second line segment of the pattern feature is angled in a range of 125° to 145° with respect to the first epipolar line;
causing an image sensor to capture an image of the scene including at least a portion of the light pattern;
identifying an image feature in the image, the image feature comprising two intersecting line segments that intersect at a point in the image that corresponds to the first epipolar line;
identifying at least the pattern feature as potentially corresponding to the image feature based on the image feature and the pattern feature both being associated with the first epipolar line;
estimating a plane hypothesis associated with the pattern feature based on properties of the pattern feature and properties of the image feature, wherein the plane hypothesis is associated with a set of parameters that characterize a plane; and
identifying a planar surface in the scene based on the plane hypothesis.
12. The method of claim 11 ,
wherein the two-dimensional light pattern comprises a set of N pattern features {P 1 , . . . , P N }, including the pattern feature, that are each disposed on the first epipolar line; and
wherein the method further comprises:
identifying the set of N pattern features as potentially corresponding to the image feature based on the image feature and the set of N pattern features all being associated with the first epipolar line;
estimating a first set of N plane hypotheses each associated with a respective pattern feature of the set of N pattern features, each plane hypothesis of the first set of N plane hypotheses based on properties of the image feature and properties of the respective pattern feature,
wherein each plane hypothesis of the first set of N plane hypotheses is associated with a set of parameters that characterize a plane;
identifying a second image feature in the image that is associated with the first epipolar line, the second image feature comprising two intersecting line segments that intersect at a point that corresponds to the first epipolar line;
identifying the set of N pattern features as potentially corresponding to the second image feature based on the second image feature and the set of N pattern features all being associated with the first epipolar line;
estimating a second set of N plane hypotheses each associated with a respective pattern feature of the set of N pattern features, each plane hypothesis of the second set of N plane hypotheses based on properties of the second image feature and properties of the respective pattern feature,
wherein each plane hypothesis of the second set of N plane hypotheses comprises a set of parameters that characterize a plane; and
identifying the planar surface in the scene based on a plane hypothesis that is a member of both the first set of N plane hypotheses and the second set of N plane hypotheses.
13. The method of claim 12 , wherein N >2, and a distance along the first epipolar line between pattern feature P 1 and pattern feature P 2 is different than a distance along the first epipolar line between pattern feature P N−1 and pattern feature P N , such that the set of N pattern features {P 1 , . . . , P N } are non-uniformly spaced.
14. The method of claim 13 , wherein the two-dimensional light pattern comprises a second set of N pattern features {P N+1 , . . . , P 2N } that are each disposed on a second epipolar line, and
wherein a distance along the second epipolar line between pattern feature P N1 and pattern feature P N+2 is different than a distance along the second epipolar line between pattern feature P 2N−1 and pattern feature P 2N , and
the distance along the second epipolar line between pattern feature P N1 and pattern feature P N+2 is different than a distance along the first epipolar line between pattern feature P 1 and pattern feature P 2 , such that the second set of N pattern features {P N+1 , . . . , P 2N } are non-uniformly spaced and are spaced differently than the set of N pattern features {P 1 , . . . , P N }.
15. The method of claim 11 , wherein identifying the image feature in the image comprises determining that the image feature is associated with the first epipolar line based on a proximity between the point at which the two line segments intersect and the first epipolar line.
16. The method of claim 11 ,
wherein the light source comprises a plurality of rows, and the image sensor comprises a second plurality of rows, and
the light source and the image sensor are rectified such that each row of the plurality of rows shares an epipolar line with a row of the second plurality of rows.
17. The method of claim 11 ,
wherein the properties of the pattern feature comprise a first vector u p corresponding to a first line segment of the pattern feature, a second vector v p corresponding to a second line segment of the pattern feature, and coordinates p p of the point at which the first vector u p and the second vector v p intersect, and
wherein the properties of the image feature comprise a third vector u c corresponding to a first line segment of the image feature, a fourth vector v c corresponding to a second line segment of the image feature, and coordinates p c of the point at which the third vector u c and the fourth vector v c intersect.
18. The method of claim 17 , wherein the set of parameters associated with the plane hypothesis define a plane Π and include a distance D corresponding to a shortest distance between an origin c c of a camera coordinate system to the plane Π, a polar angle θ between a plane normal vector n and a z axis extending through c c and a center of an image plane defined by the image sensor, and an azimuthal φ between the plane normal vector n and an x axis extending along a horizontal direction of the image plane.
19. The method of claim 18 , further comprising:
determining the plane normal vector n using the relationship
n
=
(
(
p
p
×
v
p
)
×
(
p
c
×
v
c
)
)
×
(
(
p
p
×
u
p
)
×
(
p
c
×
u
c
)
)
(
(
p
p
×
v
p
)
×
(
p
c
×
v
c
)
)
×
(
(
p
p
×
u
p
)
×
(
p
c
×
u
c
)
)
;
and
D
=
b
n
T
p
p
p
p
x
-
p
c
x
-
n
T
c
p
,
determining the distance D using the relationship where b is a distance between the origin c c of the camera coordinate system and an origin c p of a pattern coordinate system.
20. The method of claim 11 , further comprising:
causing the image sensor to capture a second image of the scene that does not include the light pattern;
calculating a difference image based on a difference between the image and the second image; and
identifying the image feature in the image using the difference image.
21. A non-transitory computer readable medium containing computer executable instructions that, when executed by a processor, cause the processor to perform a method for recovering planes in a scene, the method comprising:
causing a light source to emit a two-dimensional light pattern toward the scene,
wherein the two-dimensional light pattern comprises a pattern feature that is disposed on a first epipolar line, the pattern feature comprising two intersecting line segments that intersect at a point that is located on the first epipolar line, and
wherein a first line segment of the pattern feature is angled in a range of 35° to 55° with respect to the first epipolar line, and the second line segment of the pattern feature is angled in a range of 125° to 145° with respect to the first epipolar line;
causing an image sensor to capture an image of the scene including at least a portion of the light pattern;
identifying an image feature in the image, the image feature comprising two intersecting line segments that intersect at a point in the image that corresponds to the first epipolar line;
identifying at least the pattern feature as potentially corresponding to the image feature based on the image feature and the pattern feature both being associated with the first epipolar line;
estimating a plane hypothesis associated with the pattern feature based on properties of the pattern feature and properties of the image feature, wherein the plane hypothesis is associated with a set of parameters that characterize a plane; and
identifying a planar surface in the scene based on the plane hypothesis.
22. The non-transitory computer readable medium of claim 21 ,
wherein the two-dimensional light pattern comprises a set of N pattern features {P 1 , . . . , P N }, including the pattern feature, that are each disposed on the first epipolar line; and
wherein the method further comprises:
identifying the set of N pattern features as potentially corresponding to the image feature based on the image feature and the set of N pattern features all being associated with the first epipolar line;
estimating a first set of N plane hypotheses each associated with a respective pattern feature of the set of N pattern features, each plane hypothesis of the first set of N plane hypotheses based on properties of the image feature and properties of the respective pattern feature,
wherein each plane hypothesis of the first set of N plane hypotheses is associated with a set of parameters that characterize a plane;
identifying a second image feature in the image that is associated with the first epipolar line, the second image feature comprising two intersecting line segments that intersect at a point that corresponds to the first epipolar line;
identifying the set of N pattern features as potentially corresponding to the second image feature based on the second image feature and the set of N pattern features all being associated with the first epipolar line;
estimating a second set of N plane hypotheses each associated with a respective pattern feature of the set of N pattern features, each plane hypothesis of the second set of N plane hypotheses based on properties of the second image feature and properties of the respective pattern feature,
wherein each plane hypothesis of the second set of N plane hypotheses comprises a set of parameters that characterize a plane; and
identifying the planar surface in the scene based on a plane hypothesis that is a member of both the first set of N plane hypotheses and the second set of N plane hypotheses.
23. The non-transitory computer readable medium of claim 21 , wherein identifying the image feature in the image comprises determining that the image feature is associated with the first epipolar line based on a proximity between the point at which the two line segments intersect and the first epipolar line.