Fast object detection for augmented reality systems
View Patent ↗A detection method is based on a statistical analysis of the appearance of model patches from all possible viewpoints in the scene, and incorporates 3D geometry during both matching and pose estimation processes. By analyzing the computed probability distribution of the visibility of each patch from different viewpoints, a reliability measure for each patch is estimated. That reliability measure is useful for developing industrial augmented reality applications. Using the method, the pose of complex objects can be estimated efficiently given a single test image.
1. A method for real-time pose estimation of an object in a sample view, the method comprising the steps of:
selecting, in an off-line environment, a set of stable feature regions of the object;
incorporating, in an off-line environment, multiple view descriptors of a view set for each selected feature region into a statistical model;
in a computer, constraining a search area of the statistical model using geometric consistencies between the statistical model and the sample view; and
in a computer, searching the constrained search area to match regions in the statistical model with regions in the sample view;
wherein the step of constraining a search area of the statistical model using geometric consistencies between the statistical model and the sample view, further comprises matching said stable feature regions with classes of regions visible from viewpoints;
further wherein said matching is an iterative process wherein a next candidate region is chosen based on visibility from a viewpoint; and
further wherein an Nth candidate match f k ′⇄C k ′,1<k≦N is selected in a deterministic manner:
(
f
k
′
,
C
k
′
)
=
arg
max
f
k
∈
F
\
{
f
1
,
…
,
f
k
-
1
}
C
k
∈
⋂
t
=
1
k
-
1
Vc
l
′
P
(
C
k
❘
f
k
)
wherein f is a 2D feature, C is a class representing a view set, and V c 1 is a set of all classes from where C 1 is visible.
2. The method of claim 1 , wherein the step of selecting, in an off-line environment, a set of stable feature regions of the object, further comprises the steps of:
creating a set of environmental maps of the object;
extracting affine covariant features from the environmental maps; and
selecting a set of feature regions based on a stability of a feature across multiple environmental maps.
3. The method of claim 2 , wherein the step of selecting a set of feature regions based on a stability of a feature across multiple environmental maps, further comprises:
comparing the feature across multiple environmental maps using a Mahalanobis distance of respective SIFT descriptors.
4. The method of claim 3 , further comprising the step of:
removing repetitive feature regions by clustering feature regions based on similarity.
5. The method of claim 1 , wherein the step of incorporating, in an off-line environment, multiple view descriptors of a view set for each selected feature region into a statistical model, further comprises the steps of:
determining SIFT descriptors of the selected feature regions;
reducing a dimensionality of gradient image vectors of the descriptors using PCA-SIFT;
minimizing an impact of illumination by normalizing the vectors; and
ensuring a Gaussian distribution of the gradient image vectors by applying a Chi-square test.
6. A method for real-time pose estimation of an object in a sample view, the method comprising the steps of:
selecting, in an off-line environment, a set of stable feature regions of the object;
incorporating, in an off-line environment, multiple view descriptors of a view set for each selected feature region into a statistical model;
in a computer, constraining a search area of the statistical model using geometric consistencies between the statistical model and the sample view; and
in a computer, searching the constrained search area to match regions in poses in the statistical model with regions in the sample view, including:
extracting feature regions from the sample view;
projecting multiple view descriptors of a view set for each feature region extracted from the sample view into a feature space; and
processing each view descriptor using a Bayesian classifier to determine whether it belongs to a view set class of the statistical model;
wherein the step of processing each view descriptor using a Bayesian classifier, further comprises the steps of:
selecting a match by determining an a posteriori probability, P(C i |f j ), whether a test feature f j belongs to a view set class C i
P
(
C
i
f
j
)
=
P
(
f
j
❘
C
i
)
P
(
C
i
)
∑
k
=
1
N
p
(
f
j
❘
C
k
)
P
(
C
k
)
;
and
processing each view descriptor using a Bayesian classifier to determine whether it belongs to the view set class of the statistical model.
7. The method of claim 6 , wherein the step of processing each view descriptor using a Bayesian classifier includes using a threshold probability to accelerate runtime matching.
8. The method of claim 6 , wherein the step of selecting, in an off-line environment, a set of stable feature regions of the object, further comprises the steps of:
creating a set of environmental maps of the object;
extracting affine covariant features from the environmental maps; and
selecting a set of feature regions based on a stability of a feature across multiple environmental maps.
9. The method of claim 8 , wherein the step of selecting a set of feature regions based on a stability of a feature across multiple environmental maps, further comprises:
comparing the feature across multiple environmental maps using a Mahalanobis distance of respective SIFT descriptors.
10. The method of claim 9 , further comprising the step of:
removing repetitive feature regions by clustering feature regions based on similarity.
11. The method of claim 6 , wherein the step of incorporating, in an off-line environment, multiple view descriptors of a view set for each selected feature region into a statistical model, further comprises the steps of:
determining SIFT descriptors of the selected feature regions;
reducing a dimensionality of gradient image vectors of the descriptors using PCA-SIFT;
minimizing an impact of illumination by normalizing the vectors; and
ensuring a Gaussian distribution of the gradient image vectors by applying a Chi-square test.
12. A computer program product comprising a computer readable recording medium having recorded thereon a computer program comprising code means for, when executed on a computer, instructing said computer to control steps in a method for real-time pose estimation of an object in a sample view, the method comprising the steps of:
selecting, in an off-line environment, a set of stable feature regions of the object;
incorporating, in an off-line environment, multiple view descriptors of a view set for each selected feature region into a statistical model;
constraining a search area of the statistical model using geometric consistencies between the statistical model and the sample view; and
searching the constrained search area to match regions in the statistical model with regions in the sample view;
wherein the step of constraining a search area of the statistical model using geometric consistencies between the statistical model and the sample view, further comprises matching said stable feature regions with classes of regions visible from viewpoints;
further wherein said matching is an iterative process wherein a next candidate region is chosen based on visibility from a viewpoint; and
further wherein an Nth candidate match f k ′⇄C k ′,1<k≦N is selected in a deterministic manner:
(
f
k
′
,
C
k
′
)
=
arg
max
f
k
∈
F
\
{
f
1
,
…
,
f
k
-
1
}
C
k
∈
⋂
t
=
1
k
-
1
Vc
l
′
P
(
C
k
f
k
)
wherein f is a 2D feature, C is a class representing a view set, and V C l is a set of all classes from where C l is visible.
13. The computer program product of claim 12 , wherein the step of selecting, in an off-line environment, a set of stable feature regions of the object, further comprises the steps of:
creating a set of environmental maps of the object;
extracting affine covariant features from the environmental maps; and
selecting a set of feature regions based on a stability of a feature across multiple environmental maps.
14. The computer program product of claim 12 , wherein the step of selecting a set of feature regions based on a stability of a feature across multiple environmental maps, further comprises:
comparing the feature across multiple environmental maps using a Mahalanobis distance of respective SIFT descriptors.
15. The computer program product of claim 14 , wherein the method further comprises the step of:
removing repetitive feature regions by clustering feature regions based on similarity.
16. The computer program product of claim 12 , wherein the step of incorporating, in an off-line environment, multiple view descriptors of a view set for each selected feature region into a statistical model, further comprises the steps of:
determining SIFT descriptors of the selected feature regions;
reducing a dimensionality of gradient image vectors of the descriptors using PCA-SIFT;
minimizing an impact of illumination by normalizing the vectors; and
ensuring a Gaussian distribution of the gradient image vectors by applying a Chi-square test.
17. A computer program product comprising a computer readable recording medium having recorded thereon a computer program comprising code means for, when executed on a computer, instructing said computer to control steps in a method for real-time pose estimation of an object in a sample view, the method comprising the steps of:
selecting, in an off-line environment, a set of stable feature regions of the object;
incorporating, in an off-line environment, multiple view descriptors of a view set for each selected feature region into a statistical model;
constraining a search area of the statistical model using geometric consistencies between the statistical model and the sample view; and
searching the constrained search area to match regions in the statistical model with regions in the sample view, including:
extracting feature regions from the sample view;
projecting multiple view descriptors of a view set for each feature region extracted from the sample view into a feature space; and
processing each view descriptor using a Bayesian classifier to determine whether it belongs to a view set class of the statistical model;
wherein the step of processing each view descriptor using a Bayesian classifier, further comprises the steps of:
selecting a match by determining an a posteriori probability, P(C i |f j ), whether a test feature f j belongs to a view set class C i
P
(
C
i
f
j
)
=
P
(
f
j
❘
C
i
)
P
(
C
i
)
∑
k
=
1
N
p
(
f
j
❘
C
k
)
P
(
C
k
)
;
and
processing each view descriptor using a Bayesian classifier to determine whether it belongs to the view set class of the statistical model.
18. The computer program product of claim 17 , wherein the step of processing each view descriptor using a Bayesian classifier includes using a threshold probability to accelerate runtime matching.
19. The computer program product of claim 17 , wherein the step of selecting, in an off-line environment, a set of stable feature regions of the object, further comprises the steps of:
creating a set of environmental maps of the object;
extracting affine covariant features from the environmental maps; and
selecting a set of feature regions based on a stability of a feature across multiple environmental maps.
20. The computer program product of claim 19 , wherein the step of selecting a set of feature regions based on a stability of a feature across multiple environmental maps, further comprises:
comparing the feature across multiple environmental maps using a Mahalanobis distance of respective SIFT descriptors.
21. The computer program product of claim 20 , wherein the method further comprises the step of:
removing repetitive feature regions by clustering feature regions based on similarity.
22. The computer program product of claim 17 , wherein the step of incorporating, in an off-line environment, multiple view descriptors of a view set for each selected feature region into a statistical model, further comprises the steps of:
determining SIFT descriptors of the selected feature regions;
reducing a dimensionality of gradient image vectors of the descriptors using PCA-SIFT;
minimizing an impact of illumination by normalizing the vectors; and
ensuring a Gaussian distribution of the gradient image vectors by applying a Chi-square test.