Systems and methods for feature fusion

1. A method comprising:

defining initial inter-visual word relationships between a plurality of visual words that were generated based on visual features;

defining visual word-image relationships between the plurality of visual words and a plurality of images;

defining inter-image relationships between the plurality of images;

generating revised inter-visual word relationships in a vector space based on the initial inter-visual word relationships, the inter-image relationships, and the visual word-image relationships, wherein generating the revised inter-visual word relationships in the vector space based on the initial inter-visual word relationships, the inter-image relationships, and the visual word-image relationships includes

generating a diffusion map that represents the initial inter-visual word relationships, the inter-image relationships, and the visual word-image relationships,

wherein the diffusion map includes nodes,

wherein the nodes represents a respective visual word or a respective image, and

wherein a diffusion distance between two nodes in the diffusion map is based on a likelihood that a Markov chain transits from each of the two nodes to a same node by following any arbitrary path that has a length of a given number of transition steps; and

generating higher-level visual words in the vector space based on the revised inter-visual word relationships.

2. The method of claim 1 , wherein generating the revised inter-visual word relationships includes generating a weight matrix based on the initial inter-visual word relationships, the visual word-image relationships, and the inter-image relationships.

3. The method of claim 1 , wherein the initial inter-visual word relationships are based on a visual similarity of the respective visual words, and wherein the inter-image relationships are based on a visual similarity of the respective images.

4. The method of claim 1 , wherein the visual word-image relationships are based on co-occurrences of respective visual words and images.

5. The method of claim 1 , wherein generating the revised inter-visual word relationships in the vector space based on the initial inter-visual word relationships, the inter-image relationships, and the visual word-image relationships includes mapping the initial inter-visual word relationships, the inter-image relationships, and the visual word-image relationships to the vector space.

6. The method of claim 5 , wherein the revised inter-visual word relationships in the vector space are based on respective Euclidian distances in the vector space.

7. The method of claim 6 , further comprising generating clusters of visual words in the vector space based on the revised inter-visual word relationships, wherein the higher-level visual words are generated based on the clusters of visual words.

8. The method of claim 1 , further comprising:

defining semantic label-visual word relationships between a plurality of semantic labels and the plurality of visual words; and

defining inter-semantic label relationships between the semantic labels,

wherein the revised inter-visual word relationships in the vector space are generated further based on the semantic label-visual word relationships and the inter-semantic label relationships.

9. The method of claim 8 , wherein the inter-semantic label relationships are defined according to a hierarchy.

10. The method of claim 1 , wherein the visual words were generated in a native visual-feature space of the visual features, and wherein a visual word encompasses a space in the native visual-feature space.

11. The method of claim 10 , wherein a higher-level visual word includes a group of two or more visual words.

12. The method of claim 1 , wherein the diffusion distance is calculated according to

[

D

(

t

)

⁡

(

z

i

,

z

j

)

]

2

=

∑

q

∈

Z

⁢

(

p

iq

(

t

)

-

p

jq

(

t

)

)

2

φ

⁡

(

z

q

)

(

0

)

,

where D (t) (z i ,z j ) is the diffusion distance; where z i , z j , and z q are nodes in the diffusion map; where p iq (t) and p jq (t) are random-walk forward probabilities; and where φ(z q ) (0) is a stationary distribution that indicates a density of the visual words.

13. The method of claim 1 , wherein the diffusion distance is calculated according to

[

D

(

t

)

⁡

(

z

i

,

z

j

)

]

2

≈

∑

s

=

2

d

+

1

⁢

(

λ

s

t

)

2

⁢

(

φ

s

⁡

(

z

i

)

-

φ

s

⁡

(

z

j

)

)

2

,

where D (t) (z i ,z j ) is the diffusion distance, where z i and z j are nodes in the diffusion map, where φ s (z i ) and φ s (z j ) are eigenvectors of a transition matrix, and where λ s t is an eigenvalue.

14. The method of claim 1 , wherein some visual words in the plurality of visual words are a first type of visual word, which are visual words that were generated based on visual features that are a first type of visual feature, and wherein some visual words in the plurality of visual words are a second type of visual word, which are visual words that were generated based on visual features that are a second type of visual feature, and

wherein the visual words that are the first type of visual word have a different native visual-feature space than the visual words that are the second type of visual word.

15. The method of claim 14 , wherein the initial inter-visual word relationships, the inter-image relationships, and the visual word-image relationships can be represented by a weight matrix W, wherein

W

=

(

II

AI

T

SI

T

AI

AA

AS

T

SI

AS

SS

)

,

where II denotes inter-image relationships, where AA denotes the inter-visual word relationships of the visual words that are the first type of visual word, where SS denotes the inter-visual word relationships of the visual words that are the second type of visual word, where AI denotes the visual word-image relationships of the images and of the visual words that are the first type of visual word, where SI denotes the visual word-image relationships of the images and of the visual words that are the second type of visual word, and where AS denotes the inter-visual word relationships of the visual words that are the first type of visual word and the visual words that are the second type of visual word.

16. A method comprising:

defining initial inter-visual word relationships between a plurality of visual words that were generated based on visual features;

defining semantic label-visual word relationships between a plurality of semantic labels and the plurality of visual words;

defining inter-semantic label relationships between the plurality of semantic labels;

generating revised inter-visual word relationships in a vector space based on the initial inter-visual word relationships, the semantic label-visual word relationships, and the inter-semantic label relationships, wherein generating the revised inter-visual word relationships in the vector space based on the initial inter-visual word relationships, the semantic label-visual word relationships, and the inter-semantic label relationships includes

generating a diffusion map that represents the initial inter-visual word relationships, the semantic label-visual word relationships, and the inter-semantic label relationships,

wherein the diffusion map includes nodes,

wherein the nodes represents a respective visual word or a respective semantic label, and

wherein a diffusion distance between two nodes in the diffusion map is based on a likelihood that a Markov chain transits from each of the two nodes to a same node by following any arbitrary path that has a length of a given number of transition steps; and

generating higher-level visual words in the vector space based on the revised inter-visual word relationships.

17. The method of claim 16 , further comprising:

defining visual word-image relationships between the plurality of visual words and the plurality of images,

wherein the revised inter-visual word relationships in the vector space are generated further based on the visual word-image relationships.

18. One or more non-transitory computer-readable media storing instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising:

defining initial inter-visual word relationships between a plurality of visual words that were generated based on visual features;

defining semantic label-visual word relationships between a plurality of semantic labels and the plurality of visual words;

defining inter-semantic label relationships between the plurality of semantic labels;

generating revised inter-visual word relationships in a vector space based on the initial inter-visual word relationships, the semantic label-visual word relationships, and the inter-semantic label relationships, wherein generating the revised inter-visual word relationships in the vector space based on the initial inter-visual word relationships, the semantic label-visual word relationships, and the inter-semantic label relationships includes

generating a diffusion map that represents the initial inter-visual word relationships, the semantic label-visual word relationships, and the inter-semantic label relationships,

wherein the diffusion map includes nodes,

wherein each of the nodes represents a respective visual word or a respective semantic label, and

wherein a diffusion distance between two nodes in the diffusion map is based on a likelihood that a Markov chain transits from each of the two nodes to a same node by following any arbitrary path that has a length of a given number of transition steps; and

generating higher-level visual words in the vector space based on the revised inter-visual word relationships.

19. A system comprising:

one or more computer-readable media configured to store features; and

one or more processors that are coupled to the one or more computer-readable media and that are configured to cause the system to

define initial inter-visual word relationships between a plurality of visual words that were generated based on visual features,

define visual word-image relationships between the plurality of visual words and a plurality of images,

define inter-image relationships between the plurality of images,

generate revised inter-visual word relationships in a vector space based on the initial inter-visual word relationships, the visual word-image relationships, and the inter-image relationships, wherein, to generate the revised inter-visual word relationships in the vector space, the one or more processors are configured to cause the system to

generate a diffusion map that represents the initial inter-visual word relationships, the inter-image relationships, and the visual word-image relationships,

wherein the diffusion map includes nodes,

wherein the nodes represents a respective visual word or a respective image, and

wherein a diffusion distance between two nodes in the diffusion map is based on a likelihood that a Markov chain transits from each of the two nodes to a same node by following any arbitrary path that has a length of a given number of transition steps, and

generate higher-level visual words in the vector space based on the revised inter-visual word relationships.