Fine-grained image classification by exploring bipartite-graph labels
Systems and methods are disclosed for deep learning and classifying images of objects by receiving images of objects for training or classification of the objects; producing fine-grained labels of the objects; providing object images to a multi-class convolutional neural network (CNN) having a softmax layer and a final fully connected layer to explicitly model bipartite-graph labels (BGLs); and optimizing the CNN with global back-propagation.
1. A method for deep learning and classifying images of objects, comprising:
receiving a set of training images of objects;
producing k fine-grained labels of the objects;
providing the labeled set of training images to a multi-class convolutional neural network (CNN) having a softmax layer and a final fully connected layer to explicitly model bipartite-graph labels (BGLs), and for m types of coarse classes, where each coarse type j (for j=1, . . . , m) contains k j coarse classes, modeling BGL relations of the k fine-grained classes with the coarse classes as m bipartite graphs grouped in a star-like structure;
optimizing the CNN with global back-propagation; and
classifying input images of objects into fined-grained classes using the optimized CNN.
2. The method of claim 1 , comprising modeling a relation between the fine-grained labels and attributes in a hierarchical manner.
3. The method of claim 1 , comprising applying the BGLs to jointly model fine-grained classes with predefined coarse classes.
4. The method of claim 3 , comprising grouping fine-grained classes as a coarse class.
5. The method of claim 1 , for each graph of coarse type j, comprising encoding a bipartite structure in a binary association matrix G j ∈{0,1} k×k j , whose element g ic j j =1 if the i-th fine-grained label is connected with coarse label c j .
6. The method of claim 1 , comprising:
generating scores f=W T x∈ k for k fine-grained classes;
generating scores f j =W j T x∈ k j for k j coarse classes of type j;
augmenting a last fully-connected layer W with m additional variables, {W j } j=1 m and
with an input image x of i-th fine-gained class, modeling a BGL joint probability with m coarse labels {c j } j=1 m as,
P
(
i
,
{
c
j
}
j
❘
x
,
W
,
{
W
j
}
j
)
=
1
z
e
f
i
∏
j
=
1
m
g
ic
j
j
e
f
c
j
j
where the z is the partition function computed as,
z
=
∑
i
=
1
k
e
f
i
∏
j
=
1
m
∑
c
j
=
1
k
j
g
ic
j
j
e
f
c
j
j
.
7. The method of claim 1 , comprising optimizing the BGL using back-propagation with mini-batch stochastic gradient descent.
8. The method of claim 7 , comprising determining gradients for each parameter in closed-form.
9. The method of claim 8 , comprising determining the gradients as:
∂
log
p
y
∂
f
i
=
1
[
i
=
y
]
-
p
i
,
∂
log
p
y
∂
f
c
j
j
=
1
[
g
yc
j
j
=
1
]
-
p
c
j
j
,
∂
log
p
ϕ
y
j
j
∂
f
i
=
p
i
p
ϕ
y
j
j
1
[
g
i
ϕ
y
j
j
=
1
]
-
p
i
,
∂
log
p
ϕ
y
j
j
∂
f
c
j
j
=
1
[
c
j
=
ϕ
y
j
]
-
p
c
j
j
,
∂
log
p
ϕ
y
j
j
∂
f
c
l
l
=
∑
i
=
1
k
g
i
ϕ
y
j
j
g
il
l
p
i
p
ϕ
y
j
j
-
p
c
l
l
,
l
≠
j
,
∂
log
p
w
∂
w
i
=
-
λ
∑
j
=
1
m
∑
c
j
=
1
k
j
g
ic
j
j
(
w
i
-
w
c
j
j
)
,
∂
log
p
w
∂
w
c
j
j
=
-
λ
∑
i
=
1
k
g
ic
j
j
(
w
c
j
j
-
w
i
)
,
where ϕ y j =c y denotes the non-zero element of i-th row of G j , that is, g ic j j =1; p i and p ϕ y j j are the marginal probability derived from claim 7 ; w i is i-th column of W; w c j j is c j -column of W j .
10. The method of claim 1 , wherein the objects comprises a finished good, further comprising modeling label structures between fine-grained labels and their ingredients in a bipartite graph.
11. The method of claim 10 wherein the finished good comprises food.
12. The method of claim 1 , comprising classifying food type and restaurant source.
13. The method of claim 1 , comprising modeling a bi-partite graph structure in a conditional random field (CRF) and optimizing lower-level features and high-level connections in the CNN.
14. The method of claim 1 , for a training data X and a graph label defined by {G j } j=1 m , comprising determining a last layer of the CNN with BGL to minimize a joint negative log-likelihood with proper regularization over weights:
W
,
{
W
j
}
j
min
∑
(
x
,
y
)
∈
χ
(
-
log
p
y
-
∑
j
=
1
m
log
p
ϕ
y
j
j
)
-
log
p
w
,
see claim 10 for the definition of each component.
15. The method of claim 1 , comprising generating bipartite-graph labels for fine-grained object recognition in the CNNs and modeling a bi-partite graph structure in a conditional random fields (CRF) and optimizing low-level features and high-level connections in the CNN.
16. A system to perform deep learning and classifying images of objects, comprising:
a processor;
an image sensor coupled to the processor, the image sensor capturing images of objects;
a multi-class convolutional neural network (CNN) having a softmax layer and a final fully connected layer to explicitly model bipartite-graph labels (BGLs); and
a computer readable module coupled to the CNN for:
receiving a set of training images of objects;
producing k fine-grained labels of the objects;
providing the labeled set of training images to the multi-class convolutional neural network (CNN) having a softmax layer and a final fully connected layer to explicitly model bipartite-graph labels (BGLs), and for m types of coarse classes, where each coarse type j (for j=1, . . . , m) contains k j coarse classes, modeling BGL relations of the k fine-grained classes with the coarse classes as m bipartite graphs grouped in a star-like structure;
optimizing the CNN with global back-propagation; and
classifying input images of objects into fined-grained classes using the optimized CNN.
17. The system of claim 16 , comprising code for modeling a relation between the fine-grained labels and attributes in a hierarchical manner.
18. The system of claim 16 , comprising code for applying the BGLs to jointly model fine-grained classes with predefined coarse classes.
19. The system of claim 16 , comprising code for:
generating scores f=W T x∈ k for k fine-grained classes;
generating scores f j =W j T x∈ k j for k j coarse classes of type j;
augmenting a last fully-connected layer W with m additional variables, {W j } j=1 m and
with an input image x of i-th fine-gained class, modeling a BGL joint probability with m coarse labels {c j } j=1 m as,
P
(
i
,
{
c
j
}
j
❘
x
,
W
,
{
W
j
}
j
)
=
1
z
e
f
i
∏
j
=
1
m
g
ic
j
j
e
f
c
j
j
.