Automatically evaluating summarizers

1. A method, comprising:

receiving content to be summarized;

analyzing the content using an extractive summarizer to determine a reference extractive summary of the content;

analyzing the content using a plurality of different abstractive summarizers to determine candidate abstractive summaries of the content;

automatically comparing each of the candidate abstractive summaries with the reference extractive summary to determine corresponding evaluation metrics for the candidate abstractive summaries with respect to the reference extractive summary; and

based at least in part on the evaluation metrics that indicate relative performances of the different abstractive summarizers with respect to the extractive summarizer, automatically selecting one of the candidate abstractive summaries as a selected summary to be provided.

2. The method of claim 1 , further comprising:

selecting a corresponding abstractive summarizer from the plurality of different abstractive summarizers associated with the selected summary; and

configuring the selected abstractive summarizer for summarizing a second content received to be summarized.

3. The method of claim 2 , wherein the content received to be summarized and the second content received to be summarized are associated with a same document type.

4. The method of claim 3 , wherein the document type is a knowledge-based article, a news article, a research paper, a marketing document, a financial document, a patent document, or an email document.

5. The method of claim 1 , further comprising:

ranking the determined corresponding evaluation metrics in an order; and

associating each of the determined corresponding evaluation metrics with a corresponding one of the plurality of the different abstractive summarizers.

6. The method of claim 1 , wherein selecting one of the candidate abstractive summaries as the selected summary includes identifying a best determined evaluation metric of the determined corresponding evaluation metrics.

7. The method of claim 6 , wherein the determined corresponding evaluation metrics include Rouge-L f-measure metrics.

8. The method of claim 1 , further comprising:

averaging the determined corresponding evaluation metrics with previously determined evaluation metrics associated with the plurality of different abstractive summarizers, wherein the previously determined evaluation metrics were determined utilizing previously received content different from the received content to be summarized.

9. The method of claim 8 , wherein the previously received content and the content received to be summarized are associated with a same document type.

10. The method of claim 8 , wherein averaging the determined corresponding evaluation metrics with the previously determined evaluation metrics includes utilizing a running average to more heavily weight the determined corresponding evaluation metrics than the previously determined evaluation metrics.

11. A system, comprising:

one or more processors;

a network communication interface, wherein the network communication interface is connected to an intranet environment; and

a memory coupled to the one or more processors, wherein the memory is configured to provide the one or more processors with instructions which when executed cause the one or more processors to:

receive content to be summarized;

analyze the content using an extractive summarizer to determine a reference extractive summary of the content;

analyze the content using a plurality of different abstractive summarizers to determine candidate abstractive summaries of the content;

automatically compare each of the candidate abstractive summaries with the reference extractive summary to determine corresponding evaluation metrics for the candidate abstractive summaries with respect to the reference extractive summary; and

based at least in part on the evaluation metrics that indicate relative performances of the different abstractive summarizers with respect to the extractive summarizer, automatically select one of the candidate abstractive summaries as a selected summary to be provided.

12. The system of claim 11 , wherein the memory is further configured to provide the one or more processors with instructions which when executed cause the one or more processors to:

select a corresponding abstractive summarizer from the plurality of different abstractive summarizers associated with the selected summary; and

configure the selected abstractive summarizer for summarizing a second content received to be summarized.

13. The system of claim 12 , wherein the content received to be summarized and the second content received to be summarized are associated with a same document type.

14. The system of claim 13 , wherein the document type is a knowledge-based article, a news article, a research paper, a marketing document, a financial document, a patent document, or an email document.

15. The system of claim 11 , wherein the memory is further configured to provide the one or more processors with instructions which when executed cause the one or more processors to:

rank the determined corresponding evaluation metrics in an order; and

associate each of the determined corresponding evaluation metrics with a corresponding one of the plurality of the different abstractive summarizers.

16. The system of claim 11 , wherein selecting one of the candidate abstractive summaries as the selected summary includes identifying a best determined evaluation metric of the determined corresponding evaluation metrics.

17. The system of claim 16 , wherein the determined corresponding evaluation metrics include Rouge-L f-measure metrics.

18. The system of claim 11 , wherein the memory is further configured to provide the one or more processors with instructions which when executed cause the one or more processors to:

average the determined corresponding evaluation metrics with previously determined evaluation metrics associated with the plurality of different abstractive summarizers, wherein the previously determined evaluation metrics were determined utilizing previously received content different from the received content to be summarized.

19. The system of claim 18 , wherein the previously received content and the content received to be summarized are associated with a same document type.

20. A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

receiving content to be summarized;

analyzing the content using an extractive summarizer to determine a reference extractive summary of the content;

analyzing the content using a plurality of different abstractive summarizers to determine candidate abstractive summaries of the content;

automatically comparing each of the candidate abstractive summaries with the reference extractive summary to determine corresponding evaluation metrics for the candidate abstractive summaries with respect to the reference extractive summary; and

based at least in part on the evaluation metrics that indicate relative performances of the different abstractive summarizers with respect to the extractive summarizer, automatically selecting one of the candidate abstractive summaries as a selected summary to be provided.