Region description and modeling for image subscene recognition
A method and apparatus is described here that categorizes images by extracting regions and describing the regions with a 16-dimensional subscene feature vector, which is a concatenation of color, texture, and spatial feature vectors. By comparing the spatial feature vectors in images with similarly-obtained feature vectors in a Gaussian mixture based model pool (obtained in a subscene modeling phase), the images may be categorized (in a subscene recognition phase) with probabilities relating to each region or subscene. Higher probabilities are likelier correlations. The device may be a single or multiple core CPU, or parallelized vector processor for characterizing many images. The images may be photographs, videos, or video stills, without restriction. When used real-time, the method may be used for visual searching or sorting.
1. An image classification apparatus, comprising:
a computer; and
programming executable on said computer for classifying an input image by performing steps comprising:
(a) forming a concatenated 6D texture feature vector texture ({right arrow over (V)} texture ) as a sum of a normalized number of 45 degree and 135 degree (N 45 +N 135 ) edge points from a sum of 45° and 135° edge points;
(b) performing subscene modeling by:
(i) segmenting an input image training set from an image database into one or more segmented regions based on an input pool of N subscene categories;
(ii) extracting a training region feature vector for each of the segmented regions corresponding to the input pool of N subscene categories; and
(iii) forming a subscene model pool comprising training region feature vectors for each for the input pool of N subscene categories; and
(c) performing subscene recognition by:
(i) segmenting the input image into a set of input regions;
(ii) extracting a region feature corresponding to each of the set of input regions to form a set of extracted region feature vectors;
(iii) filtering the set of extracted region features against the subscene model pool to form a filtered output;
(iv) setting a subscene to one of a group consisting of: the training region feature vector and the set of extracted region feature vectors; and
(v) describing the subscene by a concatenated subscene feature vector {right arrow over (V)} subscene comprising:
a color feature vector {right arrow over (V)} color ;
the texture feature vector {right arrow over (V)} texture ; and
a spatial feature vector {right arrow over (V)} spatial ;
wherein the subscene is in a YCbCr color space with a luminance Y channel, a blue chrominance Cb channel, and a red chrominance Cr channel;
(vi) wherein the color feature vector {right arrow over (V)} color is calculated in the subscene using steps comprising:
calculating a mean value of the luminance Y channel and mean values blue Cb and red Cr respectively for both the blue Cb and red Cr chrominance channels;
calculating a two dimensional (2D) chrominance histogram of both the blue Cb and red Cr chrominance channels, wherein the histogram comprises bins;
calculating a subscene luminance variance as:
Var
(
Y
)
=
1
N
∑
i
=
1
N
(
Y
i
-
Y
_
)
2
;
and
calculating a subscene chrominance variance as:
Var
(
CbCr
)
=
1
N
(
∑
i
=
1
N
(
Cb
i
-
Cb
_
)
2
+
(
Cr
i
-
Cr
_
)
2
)
;
where N denotes a number of pixels in the subscene, Y i , Cb i , and Cr i , denote a pixel value in position i in the subscene in each of the luminance Y channel, and the blue Cb and red Cr chrominance channels, Y represents the mean value of Y, Cb represents the mean value of Cb, and Cr represents the mean value of Cr .
2. The apparatus of claim 1 , wherein the computer is capable of classifying an image either sequentially or at least partially in parallel.
3. The apparatus of claim 1 , wherein said subscene recognition further comprises selecting the set of subscene categories corresponding to one or more of the largest filtered outputs.
4. The apparatus of claim 1 , wherein the color feature vector {right arrow over (V)} color is calculated in the subscene using steps further comprising:
setting a largest valued of the (2D) chrominance histogram bin's corresponding Cb and Cr values respectively as dominant values D Cb and D Cr .
5. The apparatus of claim 4 , wherein the color feature vector {right arrow over (V)} color is calculated in the subscene using steps further comprising:
calculating a colorfulness value as Colorfulness =|D Cb −D Cr |.
6. The apparatus of claim 5 , wherein the color feature vector {right arrow over (V)} color is calculated for the subscene using steps further comprising:
forming a 6 dimension (6D) color feature vector ({right arrow over (V)} color ) as
{right arrow over (V)} color =( Y ,Var( Y ),Var( CbCr ), D Cb ,D Cr ,Colorfulness).
7. The apparatus of claim 6 , wherein the texture feature vector {right arrow over (V)} texture is calculated in the subscene using steps comprising:
performing an edge detection procedure on the luminance channel (Y domain) to generate a set of edge points;
categorizing each edge point into a horizontal, vertical, 45°, or 135° edge, non-directional edge, and non-edge point.
8. The apparatus of claim 7 , wherein the texture feature vector {right arrow over (V)} texture is calculated in the subscene using steps further comprising:
accumulating a sum of a normalized number of horizontal edge points (H) from the horizontal edge points;
accumulating a sum of a normalized number of vertical edge points (V) from the vertical edge points;
accumulating the sum of a normalized number (N 45 ) of 45 degree edge points from the 45° edge points;
accumulating the sum of a normalized number of 135 degree (N 135 ) edge points from the 135° edge points;
accumulating a sum of a normalized number of non-directional (N non-directional ) edge points from the non-directional edge points;
forming a sum of a normalized number of horizontal and vertical (H+V) edge points from a sum of the horizontal and vertical edge points.
9. The apparatus of claim 8 , wherein the texture feature vector {right arrow over (V)} texture is calculated using steps comprising:
forming the concatenated 6D texture feature vector ({right arrow over (V)} texture ), wherein
{right arrow over (V)} texture =( H,V ,arg max( N 45 ,N 135 ), N non-directional ,H+V,N 45 +N 135 ).
10. The apparatus of claim 9 , wherein the subscene feature vector {right arrow over (V)} subscene comprises:
{right arrow over (V)} subscene =( {right arrow over (V)} color ,{right arrow over (V)} texture ,{right arrow over (V)} spatial );
wherein {right arrow over (V)} subscene written in detail comprises:
V
→
subscene
=
(
Y
_
,
Var
(
Y
)
,
Var
(
CbCr
)
,
D
Cb
,
D
Cr
,
Colorfulness
,
H
,
V
,
argmax
(
N
45
,
N
135
)
,
N
non
-
directional
,
H
+
V
,
N
45
+
N
135
,
Y
A
,
Y
B
,
Ratio
,
Compactness
)
.
11. The apparatus of claim 1 , wherein the spatial feature vector {right arrow over (V)} spatial is calculated in the subscene using steps comprising:
forming a rectangular bounding box around the subscene;
wherein the rectangular bounding box has an upper left Y coordinate of Y A , and the lower right Y coordinate of Y B ;
calculating a Ratio of width over height of the rectangular bounding box;
calculating a Compactness of the subscene; and
forming the spatial feature vector, wherein
{right arrow over (V)} spatial =( Y A ,Y B ,Ratio,Compactness).
12. The apparatus of claim 1 , wherein said subscene modeling comprises:
selecting an optimal number of clusters in the step of forming the subscene model pool.
13. The apparatus of claim 1 , wherein said programming is stored on a computer readable medium.
14. The apparatus of claim 1 , wherein said subscene recognition is performed on a set of photographs.