Image segmentation for large-scale fine-grained recognition
View Patent ↗A method for fine-grained image classification on an image includes automatically segmenting one or more objects of interest prior to classification; and combining segmented and original image features before performing final classification.
1. A method for fine-grained image classification on an image, comprising:
automatically segmenting one or more objects of interest prior to classification;
combining segmented and original image features before performing final classification;
determining an affinity matrix W, using a feature representations f i of each pixel:
W
ij
=
exp
(
-
f
i
-
f
j
2
2
σ
2
)
;
minimizing a cost function C(X) with respect to all pixel labels X:
C
(
X
)
=
1
2
∑
i
,
j
-
1
N
W
ij
X
i
D
ii
-
X
j
D
jj
2
+
∑
i
-
1
N
λ
2
X
i
-
Y
i
2
where D ii =Σ j=1 N W ij , and Y i are the desired labels for pixels; and
solving an optimal solution for X as a system of linear equations:
((1+λ) I−S ) X=λY
X= λ((1+λ) I−S ) −1 Y.
2. The method of claim 1 , comprising obtaining initial regions with super-pixel segmentation.
3. The method of claim 1 , comprising classifying initial regions by a region model trained on the super-class as the background.
4. The method of claim 3 , comprising applying Felzenszwalb and Huttenlocher segmentation.
5. The method of claim 1 , comprising learning a model to detect one or more regions of object(s) in the image.
6. The method of claim 1 , wherein the segmentation uses Laplacian propagation applied to a foreground region and a background region.
7. The method of claim 3 , comprising reusing previously computed features in a pipeline from the image.
8. The method of claim 1 , comprising extracting high confidence regions for background and the object.
9. A method for fine-grained image classification on an original image, comprising:
automatically segmenting one or more objects of interest prior to classification by:
extracting super pixel regions from the image;
detecting high confidence regions belonging to a superclass or a background;
applying Laplacian propagation to generate a segmented image;
applying a dense grid descriptor to a Histogram of Oriented Gradients (HOG);
determining a local coordinate super-vector for the segmented image;
applying pooling to the segmented image;
combining segmented and original image features before performing final classification;
determining an affinity matrix W, using a feature representations f i of each pixel:
W
ij
=
exp
(
-
f
i
-
f
j
2
2
σ
2
)
;
minimizing a cost function C(X) with respect to all pixel labels X:
C
(
X
)
=
1
2
∑
i
,
j
-
1
N
W
ij
X
i
D
ii
-
X
j
D
jj
2
+
∑
i
-
1
N
λ
2
X
i
-
Y
i
2
where D ii =Σ j = N W ij , and Y i are the desired labels for pixels; and
solving an optimal solution for X as a system of linear equations:
((1+λ) I−S ) X=λY
X= λ((1+λ) I−S ) −1 Y.
10. The method of claim 9 , comprising classifying using a linear support vector machine (SVM).
11. The method of claim 9 , comprising learning a model to detect one or more regions of object(s) in the image.
12. The method of claim 9 , wherein the segmentation uses Laplacian propagation applied to a foreground region and a background region.
13. A system for fine-grained image classification on an image, comprising:
a camera; and
a processor coupled to the camera,
the processor comprising:
executing code for automatically segmenting one or more objects of interest prior to classification; and code for combining segmented and original image features before performing final classification;
determining an affinity matrix W, using a feature representations f i of each pixel:
W
ij
=
exp
(
-
f
i
-
f
j
2
2
σ
2
)
;
minimizing a cost function C(X) with respect to all pixel labels X:
C
(
X
)
=
1
2
∑
i
,
j
-
1
N
W
ij
X
i
D
ii
-
X
j
D
jj
2
+
∑
i
-
1
N
λ
2
X
i
-
Y
i
2
where D ii =Σ j= N W ij , and Y i are the desired labels for pixels; and
solving an optimal solution for X as a system of linear equations:
((1+λ) I−S ) X=λY
X = λ((1 +λ) I−S ) −l Y.
14. The system of claim 13 , comprising code for obtaining initial regions with super-pixel segmentation.
15. The system of claim 13 , comprising code for classifying initial regions by a region model trained on the super-class as the background.
16. The system of claim 13 , comprising code for learning a model to detect one or more regions ofobject(s) in the image.
17. The system of claim 13 , wherein the segmentation uses Laplacian propagation applied to a foreground region and a background region.
18. The system of claim 13 , comprising code for cropping the image for the segmented region within a predetermined margin to account for scale variability.