System and method for resource-adaptive, real-time new event detection

1. A method for resource adaptive, real-time new event detection in a new event detection system, comprising:

capturing, from among documents in a document streaming environment that includes a plurality of remote streaming document sources, first-story documents that mention previously unmentioned events with respect to other documents that have already been at least one of captured and processed by the system; and

dynamically adjusting a number of first-story documents captured by said capturing step by controlling a number of keywords and documents considered according to a real-time system load.

2. The method of claim 1 , wherein said capturing step comprises:

parsing the documents to determine whether any of the documents are the first-story documents; and

storing any of the documents determined to be the first-story documents in a memory.

3. The method of claim 2 , further comprising:

classifying the documents as important or non-important based on pre-specified criteria;

storing, in the memory, only information of the documents classified as important, when the memory overflows;

discarding the documents that are classified as non-important when a document consumption rate from the memory is less than a document insertion rate into the memory; and

removing the information of the documents classified as important from the memory subsequent to one of consumption and processing by a consumer.

4. The method of claim 2 , further comprising:

building at least one index that indicates, for each of the first-story documents that are stored in the memory, only a top-K most important terms from among all terms therein, a top-M most important terms, and an arrival time, where M<K;

discarding, from the memory, information about the first-story documents captured before a last W days based upon the arrival time indicated in the at least one index; and

pre-filtering, from a newly received one of the documents, older ones of the documents that are unlikely to mention a same event as the newly received one of the documents, based on the top-M most important terms in the older ones of the documents and the newly received one of the documents.

5. The method of claim 4 , wherein W, K, and M are dynamically adjusted according to the real-time system load.

6. The method of claim 2 , further comprising keeping in the memory only the first-story documents that are predicted to be cited by at least a threshold number of documents in the future, when the memory overflows.

7. The method of claim 2 , further comprising dropping from the memory at least some of the first-story documents that are cited by less than a threshold number of documents, when a document consumption rate from the memory is less than a document insertion rate into the memory.

8. The method of claim 2 , further comprising reordering the first-story documents in the memory to provide a desired one of the first-story documents to a consumer prior to other ones of the first-story documents.

9. The method of claim 2 , further comprising computing an importance of document sources using a citation relationship between the documents provided from the document sources.

10. A computer program product comprising a non-transitory computer usable medium having computer usable program code for resource adaptive, real-time new event detection in a new event detection system, said computer program product comprising:

computer usable program code for capturing, from among documents in a document streaming environment that includes a plurality of remote streaming document sources, first-story documents that mention previously unmentioned events with respect to other documents that have already been at least one of captured and processed by the system; and

computer usable program code for dynamically adjusting a number of first-story documents captured by controlling a number of keywords and documents considered according to a real-time system load.

11. The computer program product of claim 10 , further comprising:

computer usable program code for parsing the documents to determine whether any of the documents are the first-story documents; and

computer usable program code for storing any of the documents determined to be the first-story documents in a memory.

12. The computer program product of claim 11 , further comprising:

computer usable program code for classifying the documents as important or non-important based on pre-specified criteria;

computer usable program code for storing, in the memory, only information of the documents classified as important, when the memory overflows;

computer usable program code for discarding the documents that are classified as non-important when a document consumption rate from the memory is less than a document insertion rate into the memory; and

computer usable program code for removing the information of the documents classified as important from the memory subsequent to one of consumption and processing by a consumer.

13. The computer program product of claim 11 , further comprising:

computer usable program code for building at least one index that indicates, for each of the first-story documents that are stored in the memory, only a top-K most important terms from among all terms therein, a top-M most important terms, and an arrival time, where M<K;

computer usable program code for discarding, from the memory, information about the first-story documents captured before a last W days based upon the arrival time indicated in the at least one index; and

computer usable program code for pre-filtering, from a newly received one of the documents, older ones of the documents that are unlikely to mention a same event as the newly received one of the documents, based on the top-M most important terms in the older ones of the documents and the newly received one of the documents.

14. The computer program product of claim 13 , wherein W, K, and M are dynamically adjusted according to the real-time system load.

15. The computer program product of claim 11 , further comprising computer usable program code for keeping in the memory only the first-story documents that are predicted to be cited by at least a threshold number of documents in the future, when the memory overflows.

16. The computer program product of claim 11 , further comprising computer usable program code for dropping from the memory at least some of the first-story documents that are cited by less than a threshold number of documents, when a document consumption rate from the memory is less than a document insertion rate into the memory.

17. The computer program product of claim 11 , further comprising computer usable program code for reordering the first-story documents in the memory to provide a desired one of the first-story documents to a consumer prior to other ones of the first-story documents.

18. The computer program product of claim 11 , further comprising computer usable program code for computing an importance of document sources using a citation relationship between the documents provided from the document sources.

19. A resource-adaptive, real-time new event detection system, comprising:

a new event detector for capturing, from among documents in a document streaming environment that includes a plurality of remote streaming document sources, first-story documents that mention previously unmentioned events with respect to other documents that have already been at least one of captured and processed by the system; and

a non-transitory memory for storing the first-story document,

wherein a number of first-story documents captured is dynamically adjusted by controlling a number of keywords and documents considered according to a real-time system load.

20. The system of claim 19 , wherein said new event detector classifies documents as important or non-important based on pre-specified criteria, stores in the memory only information of the documents classified as important when the memory overflows, discards the documents that are classified as non-important when a document consumption rate from the memory is less than a document insertion rate into the memory, and removes the information of the documents classified as important from the memory subsequent to one of consumption and processing by a consumer.