Methods and apparatuses for identifying bilingual lexicons in comparable corpora using geometric processing

1. A method for identifying bilingual pairs in comparable corpora using a bilingual dictionary, the method comprising:

(a) using the comparable corpora to build source context vectors and target context vectors;

(b) defining: (i) a source word space with the source context vectors, and (ii) a target word space with the target context vectors;

(c) using the bilingual dictionary to project: (i) the source context vectors from the source word space to a source dictionary space, and (ii) the target context vectors from the target word space to a target dictionary space;

(d) using the source and target context vectors from the dictionary spaces to identify source/target context vector pairs in a bilingual space;

(e) computing a similarity measure for the source/target context vector pairs identified in the bilingual space to identify a one or more bilingual pairs; and

(f) adding the identified one or more bilingual pairs as entries in the bilingual dictionary;

wherein the steps (a), (b), (c), (d), (e), and (f) are each performed by a processor.

2. The method according to claim 1 , further comprising:

pruning components from the context vectors; and

normalizing the pruned context vectors.

3. The method according to claim 1 , further comprising:

building context vectors using entries in the bilingual dictionary; and

projecting at (c): (i) the source context vectors and the dictionary entry context vectors from the source word space to a source dictionary space, and (ii) the target context vectors and the dictionary entry context vectors from the target word space to a target dictionary space.

4. The method according to claim 1 , further comprising:

using the bilingual dictionary with the added one or more bilingual pairs to perform one or more of cross-language information retrieval, and cross-language categorization.

5. The method according to claim 4 , further comprising repeating (c)-(d) when the bilingual dictionary incorporates the bilingual pairs identified at (e).

6. The method according to claim 1 , wherein each context vector provides a measure of association between a selected word in the comparable corpora and other words in the comparable corpora.

7. The method according to claim 6 , where the context vector of a given word includes a set of words that co-occur in a neighborhood of the given word in the comparable corpora.

8. The method according to claim 6 , wherein the context vector of a given word includes a frequency of co-occurrence for each of a set of words in a neighborhood of the given word in the comparable corpora.

9. The method according to claim 1 , wherein (b)-(e) comprises computing similarity measure:

S ( v,w )= {right arrow over ( v )},{right arrow over ( tr ( w ))} =( P s {right arrow over (v)} ) T M ( P t {right arrow over (w)} ), where

{right arrow over (v)} is a source context vector;

{right arrow over (w)} is a target context vector;

P s is a source projection matrix;

P t is a target projection matrix;

M is a matrix that encodes the relationship between the source and target words s and t in the bilingual dictionary; and

{right arrow over (tr(w))} is the translation of the target context vector {right arrow over (w)}.

10. The method according to claim 9 , wherein the matrix M is defined as S T T, with S and T being source and target matrices that each encode relations between words and translation pairs in the bilingual dictionary (where “ T ” denotes transpose).

11. The method according to claim 1 , wherein (b)-(e) comprises computing similarity measure S(v, w) using the following equation:

S ( v,w )= SQ s {right arrow over (v)},TQ t {right arrow over (w)} ={right arrow over (v)} T Q s T S T TQ t {right arrow over (w)} , where

{right arrow over (v)} is a source context vector;

{right arrow over (w)} is a target context vector;

Q s is a projection matrix for mapping the source context vector {right arrow over (v)} on a sub-space generated by ({right arrow over (s 1 )}, . . . ,{right arrow over (s p )});

Q t is a projection matrix for mapping the target context vector {right arrow over (w)} on a sub-space generated by ({right arrow over (t 1 )}, . . . ,{right arrow over (t q )});

S is a matrix that encodes relations between source words and pairs in the bilingual dictionary;

T is a matrix that encodes relations between target words and pairs in the bilingual dictionary.

12. The method according to claim 11 , further comprising pruning translation pairs from resulting projections of context vectors {right arrow over (v)} and {right arrow over (w)} using matrices Q s and Q t , respectively.

13. The method according to claim 1 , wherein (c)-(e) comprises computing similarity measure S(v,w) using the following equation:

S ( v,w )= Ξ s {right arrow over (v)},Ξ t {right arrow over (w)} ={right arrow over (v)} T Ξ s T Ξ t {right arrow over (w)} , where

Ξ s =[ξ s 1 , . . . ,ξ s l ] T and Ξ t =[ξ t 1 , . . . ,ξ t l ] T , and

ξ s and ξ t denote directions in the source and target vector spaces.

14. The method according to claim 13 , wherein ξ s and ξ t are obtained using canonical correlation analysis.

15. The method according to claim 1 , wherein (c)-(e) comprises computing similarity measure S(v,w) using the following equation:

S ( v,w )= K ( v,w )=∇ l ( v ) T I F −1 ∇l ( w ), where

l(x)=lnP(x|θ) the log-likelihood of vocabulary words x, and

I F =E(∇l(x)∇l(y) T ) denotes the Fisher information matrix where x and y are any vocabulary word from source and target vocabularies (e 1 , . . . ,e m ) and (f 1 , . . . ,f r ) respectively.

16. The method according to claim 15 , wherein I F =E(∇l(x)∇l(y) T )≈1.

17. The method according to claim 1 , wherein (c)-(e) comprises computing a plurality of similarity measures to produce an overall similarity measure.

18. The method according to claim 1 , wherein the bilingual dictionary is initialized with a set of translation pairs automatically generated using seed.

19. An article of manufacture comprising a memory device storing computer-readable program code which when executed by a processor identifies bilingual pairs in comparable corpora using a bilingual dictionary using a method comprising:

(a) using the comparable corpora for building for each source word v a context vector {right arrow over (v)} and for each target word w a context vector {right arrow over (w)};

(b) computing source and target context vectors {right arrow over (v)}′ and {right arrow over (w)}′ projected into a sub-space formed by source {right arrow over (s)} and target {right arrow over (t)} bilingual dictionary entry context vectors; and

(c) computing a similarity measure between pairs of source words v and target words w using their context vectors {right arrow over (v)}′ and {right arrow over (w)}′ projected to a bilingual space to identify bilingual pairs.

20. The article of manufacture according to claim 19 , wherein (i) the context vector {right arrow over (v)} comprises a measure of association a(v,e) for each word e in the context of v, and (ii) the context vector {right arrow over (w)} comprises a measure of association a(w,f) for each word f in the context of w.

21. The article of manufacture according to claim 19 , further comprising repeating (b) when the bilingual dictionary incorporates the bilingual pairs identified at (c).

22. The article of manufacture according to claim 19 , further comprising pruning translation pairs from context vectors {right arrow over (v)}′ and {right arrow over (w)}′ based on a measure of similarity between source {right arrow over (v)} and target {right arrow over (w)} context vectors and source {right arrow over (s)} and target {right arrow over (t)} bilingual dictionary entry context vectors.

23. An article of manufacture comprising a memory device storing computer-readable program code which when executed by a processor identifies bilingual pairs in comparable corpora using a bilingual dictionary using a method comprising:

(a) using the comparable corpora to build source context vectors and target context vectors;

(b) defining: (i) a source word space with the source context vectors, and (ii) a target word space with the target context vectors;

(c) using the bilingual dictionary to project: (i) the source context vectors from the source word space to a source dictionary space, and (ii) the target context vectors from the target word space to a target dictionary space;

(d) using the source and target context vectors from the dictionary spaces to identify source/target context vector pairs in a bilingual space;

(e) computing a similarity measure for the source/target context vector pairs identified in the bilingual space to identify a bilingual pair.

24. A method for identifying bilingual pairs in comparable corpora using a bilingual dictionary, the method comprising:

(a) using the comparable corpora for building for each source word v a context vector {right arrow over (v)} and for each target word w a context vector {right arrow over (w)};

(b) computing source and target context vectors {right arrow over (v)}′ and {right arrow over (w)}′ projected into a sub-space formed by source {right arrow over (s)} and target {right arrow over (t)} bilingual dictionary entry context vectors;

(c) computing a similarity measure between pairs of source words v and target words w using their context vectors {right arrow over (v)}′ and {right arrow over (w)}′ projected to a bilingual space to identify bilingual pairs; and

(d) adding the identified bilingual pairs as entries in the bilingual dictionary;

wherein the steps (a), (b), (c), and (d) are each performed by a processor.

25. The method according to claim 24 , further comprising:

using the bilingual dictionary with the added bilingual pairs to perform one or more of cross-language information retrieval, and cross-language categorization.