Method for computing similarity between text spans using factored word sequence kernels
A computer implemented method and an apparatus for comparing spans of text are disclosed. The method includes computing a similarity measure between a first sequence of symbols representing a first text span and a second sequence of symbols representing a second text span as a function of the occurrences of optionally noncontiguous subsequences of symbols shared by the two sequences of symbols. Each of the symbols comprises at least one consecutive word and is defined according to a set of linguistic factors. Pairs of symbols in the first and second sequences that form a shared subsequence of symbols are each matched according to at least one of the factors.
1. A method of comparing spans of text comprising:
with a computer, computing a similarity measure between a first sequence of symbols representing a first text span and a second sequence of symbols representing a second text span as a function of the occurrences of optionally noncontiguous subsequences of symbols shared by the two sequences of symbols, wherein each of the symbols comprises at least one consecutive word, the words being enriched with linguistic information allowing them to be defined according to a set of linguistic factors, whereby pairs of symbols in the first and second sequences forming a shared subsequence of symbols are each matched according to at least one of the factors and wherein all pairs of matching symbols in a shared subsequence need not match according to the same factor.
2. The method of claim 1 , further comprising, prior to the computing, processing at least one of the first and second text spans to generate a sequence of tokens and enriching the tokens with linguistic information to form the first and second sequences of symbols.
3. The method of claim 1 , wherein the computing a similarity measure comprises computing at least one factored word sequence kernel.
4. The method of claim 3 , wherein in the factored word sequence kernel, gaps in the shared subsequences are weighted with a decay factor.
5. The method of claim 3 , wherein the factored word sequence kernel has the general form:
K
n
(
s
,
t
)
=
∑
I
,
J
λ
l
(
I
)
+
l
(
J
)
∏
k
=
1
n
A
(
s
i
k
,
t
j
k
)
where:
S and t represent the first and second sequences;
n represents a length of the shared subsequences to be used in the computation;
I and J represent two of the subsequences of size n of indices ranging on the positions of the symbols in s and t respectively;
λ represents an optional decay factor which weights gaps in non-contiguous sequences;
l (I) represents the length, as the number of symbols plus any gaps, spanned by the subsequence I in s ;
l (J) represents the length, as the number of symbols plus any gaps, spanned by the subsequence J in t, respectively; and
A(s i k ,t j k ) is a function of two symbols s i k ,t j k in the sequences s and t respectively, the function quantifying the similarity between the two symbols according to the set of factors.
6. The method of claim 5 , wherein the function A(s i k ,t j k ) is a weighted sum over factors of the similarity between factors in each pair of symbols according to the expression:
A
(
u
,
v
)
=
∑
h
=
1
p
w
h
2
k
h
(
u
[
h
]
,
v
[
h
]
)
where:
u and v are two symbols to be compared, represented each by p factors;
w h 2 ,h=1, . . . , p are the weights of the individual terms in the linear combination;
k h (u[h],v[h]) is a function measuring the similarity between the h th factor of u and the h th factor of v .
7. The method of claim 3 , wherein the factored word sequence kernel has the general form:
K
n
(
s
,
t
)
=
∑
I
,
J
λ
l
(
I
)
+
l
(
J
)
∏
k
=
1
n
(
∑
h
=
1
p
w
h
2
k
h
(
s
i
k
[
h
]
,
t
j
k
[
h
]
)
)
where
S and t represent the first and second sequences;
n represents a length of the shared subsequences to be used in the computation;
I and J represent two of the subsequences of size n of indices ranging on the positions of the symbols in s and t respectively;
λ represents an optional decay factor which weights gaps in non-contiguous sequences;
l(I) represents the length, as the number of symbols plus any gaps, spanned by the subsequence I in s;
l(J) represents the length, as the number of symbols plus any gaps, spanned by the subsequence J in t, respectively;
W h represents the weight of each factor in a set of h factors; and
k h represents the similarity score between two symbols s i k , and t j k of sequences s and t , respectively, for a given factor h.
8. The method of claim 1 , wherein the factored word sequence kernel weights at least one of the factors differently from another of the factors.
9. The method of claim 1 , wherein a plurality of the factors are selected from the group consisting of surface forms, lemmas, parts-of-speech, and morphological tags.
10. The method of claim 1 wherein there are at least three factors.
11. The method of claim 1 , wherein at least one factor comprises elements of a continuous inner-product space and the similarity is computed as the inner product between the factor elements.
12. The method of claim 1 , wherein at least one factor comprises real vectors forming latent-semantic representations of symbols.
13. The method of claim 1 , wherein the computing of the similarity measure comprises computing a plurality of similarity measures each of the similarity measures being computed for a different value of n and computing a combined similarity measure as a function of the computed similarity measures, where n represents a length of the subsequences of symbols shared by the two sequences of symbols.
14. The method of claim 13 , wherein the combined similarity measure is computed as a linear combination of the plurality of similarity measures according to the expression:
K
N
(
s
,
t
)
=
∑
n
=
1
N
μ
n
K
n
(
s
,
t
)
where:
μ n ,n=1, . . . , N is a weight of the n th single-subsequence-length measure.
15. The method of claim 1 , wherein at least one of the factors comprises elements of a countable set and the similarity of two symbols in a pair of symbols, one in each subsequence, is computed using the Kronecker delta function.
16. The method of claim 1 , wherein at least one of the factors comprises strings of characters and their similarity is computed as a decreasing function of the edit distance between the two strings.
17. The method of claim 1 , wherein at least one of the factors comprises sets of symbols and their similarity is computed using a set-similarity measure such as the Dice coefficient or the Jaccard coefficient.
18. The method of claim 1 , wherein at least one of the factors comprises sets of synonym sets from a thesaurus.
19. The method of claim 1 , wherein the second sequence comprises a set of second sequences and the method comprises, for each of the second sequences, computing of the similarity measure between the first sequence of symbols and the second sequence of symbols as a function of the occurrences of optionally noncontiguous subsequences of symbols shared by the two sequences of symbols, the method further comprising evaluating a fluency of the first text span in a natural language shared by the text sequences based on the computed similarity measures.
20. The method of claim 19 , wherein the set of second sequences comprises a set of good sequences determined to have good fluency and a set of bad sequences determined to have a poor fluency.
21. The method of claim 1 wherein the text spans comprise sentences in the same natural language.
22. A computer program product comprising a non-transitory computer-readable recording medium on which instructions are recorded, which when executed by a computer, perform a method comprising:
computing a similarity measure between a first sequence of symbols representing a first text span and a second sequence of symbols representing a second text span as a function of the occurrences of optionally noncontiguous subsequences of symbols shared by the two sequences of symbols, wherein each of the symbols comprises at least one consecutive word, the words being enriched with linguistic information allowing them to be defined according to a set of linguistic factors, whereby pairs of symbols in the first and second sequences forming a shared subsequence of symbols are each matched according to at least one of the factors and wherein all pairs of matching symbols in a shared subsequence need not match according to the same factor.
23. An apparatus for computing a similarity measure between sequences of symbols in which each symbol is defined in terms of a plurality of factors, the apparatus comprising memory which stores instructions and a processing component which executes the instructions, comprising:
a linguistic processor which takes as input a text span and enriches tokens of the text span with linguistic information to form a first sequence of symbols, each of the symbols comprising an enriched token; and
a sequence kernel computation unit which takes as input the first sequence of symbols and computes a similarity measure between the first sequence of symbols and a second sequence of symbols as a function of the occurrences of optionally noncontiguous subsequences of symbols shared by the first and second sequences of symbols, whereby pairs of symbols in shared subsequences of symbols shared by the first and second sequences are independently each matched according any one or more of the factors based on the linguistic information.
24. A method of computing a similarity measure between sequences of symbols in which each symbol is defined in terms of a plurality of factors comprising:
computing a sequence kernel in which optionally non-contiguous subsequences of the first and second sequences are compared, the kernel having the general form:
K
n
(
s
,
t
)
=
∑
I
,
J
λ
l
(
I
)
+
l
(
J
)
∏
k
=
1
n
A
(
s
i
k
,
t
j
k
)
where:
S and t represent the first and second sequences;
n represents a length of the shared subsequences to be used in the computation;
I and J represent two of the subsequences of size n of indices ranging on the positions of the symbols in s and t respectively;
λ represents an optional decay factor which weights gaps in non-contiguous sequences;
l (I) represents the length, as the number of symbols plus any gaps, spanned by the subsequence I in s;
l (J) represents the length, as the number of symbols plus any gaps, spanned by the subsequence J in t, respectively;
A(s i k ,t j k ) is a function of two symbols s i k , t j k in the sequences s and t respectively, the function quantifying the similarity between the two symbols according to the set of factors.