Annotating images
View Patent ↗Methods, systems, and apparatus, including computer program products, for generating data for annotating images automatically. In one aspect, a method includes receiving an input image, identifying one or more nearest neighbor images of the input image from among a collection of images, in which each of the one or more nearest neighbor images is associated with a respective one or more image labels, assigning a plurality of image labels to the input image, in which the plurality of image labels are selected from the image labels associated with the one or more nearest neighbor images, and storing in a data repository the input image having the assigned plurality of image labels. In another aspect, a method includes assigning a single image label to the input image, in which the single image label is selected from labels associated with multiple ranked nearest neighbor images.
1. A method of image annotation performed by a data processing apparatus, the method comprising:
receiving an input image in the data processing apparatus;
identifying one or more nearest neighbor images of the input image from among a collection of digital images stored on computer-readable media by operation of the data processing apparatus, each of the one or more nearest neighbor images being associated with a respective one or more image labels, wherein the collection of digital images comprises a plurality of reference images, and wherein identifying the one or more nearest neighbor images comprises:
calculating, for each reference image, a whole-image distance between the input image and the reference image as
1
N
∑
k
=
1
N
d
~
k
,
wherein N is a number of image features extracted from both the input image and the reference image, and {tilde over (d)} k is a weighted feature distance representing a degree of difference between a k th image feature extracted from the input image and a corresponding k th image feature extracted from the reference image, and
identifying one or more reference images closest to the input image, as measured by the whole-image distances, as the one or more nearest neighbors;
assigning a plurality of image labels to the input image, wherein the plurality of image labels are selected by the data processing apparatus from the image labels associated with the one or more nearest neighbor images;
estimating a weighting vector w that minimizes the expression
∑
l
=
1
L
log
(
1
+
exp
(
-
ω
T
d
x
1
y
l
)
)
+
λ
ω
1
,
wherein L is a number of training image pairs in a training image set, d xl is a vector containing feature distances for the l th training image pair, λ is a positive weighting parameter, and y l is +1 or −1 according to whether the l th training image pair is a pair of similar or dissimilar training images; and
storing in a digital data repository the input image having the assigned plurality of image labels.
2. The method of claim 1 , wherein y l is +1 when the l th training image pair is a pair of similar training images and y l is −1 when the l th training image pair is a pair of dissimilar training images.
3. A system comprising:
a server implemented on one or more computers and operable to perform operations comprising;
receiving an input image in the server;
identifying one or more nearest neighbor images of the input image from among a collection of digital images stored on computer-readable media by operation of the server, each of the one or more nearest neighbor images being associated with a respective one or more image labels, wherein the collection of digital images comprises a plurality of reference images, and wherein identifying the one or more nearest neighbor images comprises:
calculating, for each reference image, a whole-image distance between the input image and the reference image as
1
N
∑
k
=
1
N
d
~
k
,
wherein N is a number of image features extracted from both the input image and the reference image, and {tilde over (d)} k is a weighted feature distance representing a degree of difference between a k th image feature extracted from the input image and a corresponding k h image feature extracted from the reference image, and
identifying one or more reference images closest to the input image, as measured by the whole-image distances, as the one or more nearest neighbors;
assigning a plurality of image labels to the input image, wherein the plurality of image labels are selected by the server from the image labels associated with the one or more nearest neighbor images;
wherein the server is operable to perform operations further comprising estimating a weighting vector ω that minimizes the expression
∑
l
=
1
L
log
(
1
+
exp
(
-
ω
T
d
x
1
y
l
)
)
+
λ
ω
1
,
wherein L is a number of training image pairs in a training image set, d x1 is a vector containing feature distances for the l th training image pair, λ is a positive weighting parameter, and y l is +1 or −1 according to whether the l th training image pair is a pair of similar or dissimilar training images; and
storing in a digital data repository the input image having the assigned plurality of image labels.
4. A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:
receiving an input image in the data processing apparatus;
identifying one or more nearest neighbor images of the input image from among a collection of digital images stored on computer-readable media by operation of the data processing apparatus, each of the one or more nearest neighbor images being associated with a respective one or more image labels, wherein the collection of digital images comprises a plurality of reference images, and wherein identifying the one or more nearest neighbor images comprises:
calculating, for each reference image, a whole-image distance between the input image and the reference image as, wherein N is a number of image features extracted from both the input image and the reference image, and is a weighted feature distance representing a degree of difference between a kth image feature extracted from the input image and a corresponding kth image feature extracted from the reference image, and
identifying one or more reference images closest to the input image, as measured by the whole-image distances, as the one or more nearest neighbors;
assigning a plurality of image labels to the input image, wherein the plurality of image labels are selected by the data processing apparatus from the image labels associated with the one or more nearest neighbor images; and estimating a weighting vector ω that minimizes the expression
∑
l
=
1
L
log
(
1
+
exp
(
-
ω
T
d
x
1
y
l
)
)
+
λ
ω
1
,
wherein L is a number of training image pairs in a training image set, d xl is a vector containing feature distances for the l th training image pair, λ is a positive weighting parameter, and y l is +1 or −1 according to whether the l th training image pair is a pair of similar or dissimilar training images; and
storing in a digital data repository the input image having the assigned plurality of image labels.
5. The non-transitory computer storage medium of claim 4 , wherein y l is +1 when the l th training image pair is a pair of similar training images and y l is −1 when the l th training image pair is a pair of dissimilar training images.