Automatic evaluation of summaries
View Patent ↗A probability distribution for a reference summary of a document is determined. The probability distribution for the reference summary is then used to generate a score for a machine-generated summary of the document.
1. A computer-implemented method comprising:
a processor determining a probability distribution for a word in a plurality of reference summaries of a document as
θ
R
,
i
=
μ
p
(
w
i
|
c
)
+
∑
L
a
i
,
j
j
=
1
μ
+
∑
j
=
1
L
a
o
,
j
,
where θ R,i is the probability distribution of word w i in the plurality of reference summaries, L is a number of reference summaries in the plurality of reference summaries, a i,j is a count of the number of times word w i appears in summary j, p(w i |c) is a probability of word w i given a corpus of documents c, μ is a scaling factor greater than one, and a 0,j is a count of the number of words in summary j;
the processor determining a probability distribution for the word in a machine-generated summary as
θ
A
,
i
=
a
i
+
μ
p
(
w
i
|
c
)
a
o
+
μ
where θ A,i is the probability distribution of word w i in the machine-generated summary, a i is a count of the number of times word w i appears in the machine-generated summary, p(w i |c) is the probability of word w i given a corpus of documents c, μ is the scaling factor, and a 0 is a count of the number of words in the machine-generated summary;
the processor comparing the probability distribution for the word in the reference summaries to the probability distribution for the word in the machine-generated summary to generate a score for a machine-generated summary of the document.
2. The computer-implemented method of claim 1 further comprising determining a probability distribution for a plurality of words in the plurality of reference summaries.
3. The computer-implemented method of claim 2 wherein probability distributions for a plurality of words comprise a separate word probability distribution for each word in the machine-generated summary.
4. The computer-implemented method of claim 1 wherein comparing the probability distribution for the reference summaries to the probability distribution for the machine-generated summary to generate the score comprises using one of a group of scoring functions consisting of a Jenson-Shannon divergence and a Kullback-Leibler divergence.
5. A computer-readable storage medium having computer-executable instructions for performing steps comprising:
determining a set of word probabilities for an automatically-generated summary of a document, each word probability θ A,i determined as
θ
A
,
i
=
a
i
+
μ
p
(
w
i
|
c
)
a
o
+
μ
where θ A,i is the word probability of word w i in the automatically-generated summary, a i is a count of the number of times word w i appears in the automatically-generated summary, p(w i |c) is a probability of word w i given a corpus of documents c, μ is a scaling factor greater than one, and a 0 is a count of the number of words in the automatically-generated summary;
determining a set of word probabilities for a plurality of reference summaries of the document, each word probability θ R,i determined as
θ
R
,
i
=
μ
p
(
w
i
|
c
)
+
∑
L
a
i
,
j
j
=
1
μ
+
∑
j
=
1
L
a
o
,
j
,
where θ R,i is the word probability of word w i in the plurality of reference summaries, L is a number of reference summaries in the plurality of reference summaries, a i,j is a count of the number of times word w i appears in reference summary j, p(w i |c) is the probability of word w i given a corpus of documents c, μ is the scaling factor, and a 0,j is a count of the number of words in reference summary j;
comparing the set of word probabilities for the automatically-generated summary to the set of word probabilities for the at least one reference summary to produce a score for the automatically-generated summary.
6. The computer-readable storage medium of claim 5 wherein determining a set of word probabilities for at least one reference summary of the document comprises determining a set of word probabilities for a plurality of reference summaries of the document.
7. The computer-readable storage medium of claim 5 wherein comparing the set of word probabilities for the automatically-generated summary to the set of word probabilities for the at least one reference summary to produce a score comprises computing at least one of a group consisting of a Jensen-Shannon Divergence, a Jensen-Shannon Divergence with Smoothing, and a Kullback-Leibler Divergence with Smoothing.
8. A computer-implemented system comprising:
receiving a plurality of reference summaries for a document;
a processor using the plurality of reference summaries to determine a word probability for each of a plurality of words, each word probability θ R,i determined as
θ
R
,
i
=
μ
p
(
w
i
|
c
)
+
∑
L
a
i
,
j
j
=
1
μ
+
∑
j
=
1
L
a
o
,
j
,
where θ R,i is the word probability of word w i in the plurality of reference summaries, L is a number of reference summaries in the plurality of reference summaries, a i,j is a count of the number of times word w i appears in reference summary j, p(w i |c) is a probability of word w i given a corpus of documents c, μ is a scaling factor greater than one, and a o,j is a number of words in reference summary j;
the processor using the word probabilities determined from the reference summaries to generate a score for a machine-generated summary of the document; and
the processor storing the score.
9. The computer-implemented system of claim 8 further comprising using the machine-generated summary of the document to determine a second word probability for each of the plurality of words.
10. The computer-implemented system of claim 9 wherein using the word probabilities determined from the reference summaries to generate a score further comprises using the second word probabilities determined from the machine-generated summary with the word probabilities determined from the reference summaries to determine the score.
11. The computer-implemented system of claim 10 wherein determining a score comprises determining a Jensen-Shannon divergence between the word probabilities determined from the reference summaries and the second word probabilities determined from the machine-generated summary.