Granted Patent B2

US 7,853,544 · App. 11/286,572 · Granted Dec 14, 2010

Abstract

Claims (83)

1. A computer implemented method for identifying a set of categories for unstructured text messages and training an automated classifier therefor, the method comprising:

from a stream of the unstructured text messages captured in computer readable form, selecting a subset thereof for presentation to a user as an exploration set, the subset selected from the stream by a programmed computer, wherein the selection of the exploration set is in a generally random manner though in accord with one or more set delimiting criteria provided by the user;

via a display of the programmed computer, providing the user with both (i) a reviewable presentation of each unstructured text message selected for presentation as part of the exploration set and (ii) a flag definition and assignment interface, whereby the user defines categories for the unstructured text messages and flags at least one message of the exploration set as associated with each of the categories so defined;

via the display of the programmed computer, providing the user with a reviewable presentation of a training subset of the unstructured text messages, wherein each of the unstructured text messages of the training subset is presented together with a category selection interface whereby the user accumulates, for each of at least a subset of the categories, a respective pool of training instances from the training subset for use in training an automated classifier; and

training an automated classifier to classify individual ones of the unstructured text messages using the training subset.

2. The method of claim 1 , further comprising:

successively refining the training of the automated classifier based on successive additions, by the user using the selection interface, of further unstructured text messages to the training subset; and

evaluating the successively refined training by applying the automated classifier to an audit subset of the unstructured text messages.

3. The method of claim 2 , further comprising:

via the display, providing the user with a reviewable presentation of the audit subset of unstructured text messages, wherein each of the unstructured text messages of the audit subset is presented together with the category selection interface whereby the user accumulates, for each of at least a subset of the categories, a respective pool of audit instances from the audit subset for use in the evaluating.

4. The method of claim 3 , further comprising:

successively performing the evaluating of the automated classifier based on successive additions, by the user using the selection interface, of further unstructured text messages to the audit subset.

5. The method of claim 2 , further comprising:

forgoing further successive refining of the training, based on performance of the automated classifier against the audit subset.

6. The method of claim 1 , further comprising:

computing from the unstructured text messages of the exploration set a lexical listing of words that appear in the exploration set at frequencies greater than statistically expected based on a background corpus of textual content,

wherein the flag definition and assignment interface includes a selectable presentation of the lexical listing whereby the user may specify at least some of the categories based thereon.

7. The method of claim 1 , further comprising:

applying the trained, automated classifier to a second stream of the unstructured text messages.

8. The method of claim 7 , wherein the stream and the second stream are one of

a same stream;

initial and subsequent samples of messages from same or equivalent streams; and

different streams having statistically similar distributions of features.

9. The method of claim 7 , further comprising:

deploying the trained, automated classifier as a text categorization engine on a second programmed computer.

10. The method of claim 1 ,

wherein the user includes one or more human analysts.

11. The method of claim 1 ,

where the programmed computer includes one or more computational systems programmed to perform respective steps of the method.

12. The method of claim 1 ,

wherein the reviewable presentation of the training subset of the unstructured text messages provides a next page control that adds as training instances those unstructured text messages of a current page for which categories have been selected.

13. The method of claim 12 ,

wherein the next page control is implemented as a button.

14. The method of claim 12 , further comprising:

based on successive selections of the next page control, successively refining the training of the automated classifier based on successive additions of further unstructured text messages to the training subset.

15. A computer implemented method for identifying a set of categories for unstructured text messages and training an automated classifier therefor, the method comprising:

from a stream of the unstructured text messages captured in computer readable form, selecting a subset thereof for presentation to a user as an exploration set, the subset selected from the stream by a programmed computer;

via a display of the programmed computer, providing the user with both (i) a reviewable presentation of each unstructured text message selected for presentation as part of the exploration set and (ii) a flag definition and assignment interface, whereby the user defines categories for the unstructured text messages and flags at least one message of the exploration set as associated with each of the categories so defined;

via the display of the programmed computer, providing the user with a reviewable presentation of a training subset of the unstructured text messages, wherein each of the unstructured text messages of the training subset is presented together with a category selection interface whereby the user accumulates, for each of at least a subset of the categories, a respective pool of training instances from the training subset for use in training an automated classifier;

training an automated classifier to classify individual ones of the unstructured text messages using the training subset;

successively refining the training of the automated classifier based on successive additions, by the user using the selection interface, of further unstructured text messages to the training subset;

evaluating the successively refined training by applying the automated classifier to an audit subset of the unstructured text messages; and

via the display, providing the user with a reviewable presentation of the audit subset of unstructured text messages, wherein each of the unstructured text messages of the audit subset is presented together with the category selection interface whereby the user accumulates, for each of at least a subset of the categories, a respective pool of audit instances from the audit subset for use in the evaluating,

wherein unstructured text messages from the audit and training subsets as well as respective category selections therefor are presented in a uniform manner, such that the user is generally not aware of which unstructured text messages comprise the audit subset and which unstructured text messages comprise the training subset.

16. A computer implemented method for identifying a set of categories for unstructured text messages and training an automated classifier therefor, the method comprising:

from a stream of the unstructured text messages captured in computer readable form, selecting a subset thereof for presentation to a user as an exploration set, the subset selected from the stream by a programmed computer;

via a display of the programmed computer providing the user with both (i) a reviewable presentation of each unstructured text message selected for presentation as part of the exploration set and (ii) a flag definition and assignment interface, whereby the user defines categories for the unstructured text messages and flags at least one message of the exploration set as associated with each of the categories so defined;

via the display of the programmed computer, providing the user with a reviewable presentation of a training subset of the unstructured text messages, wherein each of the unstructured text messages of the training subset is presented together with a category selection interface whereby the user accumulates, for each of at least a subset of the categories, a respective pool of training instances from the training subset for use in training an automated classifier;

training an automated classifier to classify individual ones of the unstructured text messages using the training subset; and

successively refining the training of the automated classifier based on successive additions, by the user using the selection interface, of further unstructured text messages to the training subset; and

evaluating the successively refined training by applying the automated classifier to an audit subset of the unstructured text messages,

wherein the training subset and the audit subset are disjoint sets of unstructured text messages drawn from the exploration set in a manner generally not perceivable by the user.

17. A system comprising:

a store of unstructured text messages captured in computer readable form;

a computer coupled to the store and programmed to select and present a subset of the unstructured text messages to a user as an exploration set;

the programmed computer providing the user with both (i) a reviewable presentation of each unstructured text message selected for presentation as part of the exploration set and (ii) a flag definition and assignment interface, whereby the user defines categories for the unstructured text messages and flags at least one message of the exploration set as associated with each of the categories so defined;

the programmed computer further providing the user with a reviewable presentation of a training subset of the unstructured text messages, wherein each of the unstructured text messages of the training subset is presented together with a category selection interface whereby the user accumulates, for each of at least a subset of the categories, a respective pool of training instances from the training subset for use in training an automated classifier;

the programmed computer training an automated classifier to classify individual ones of the unstructured text messages using the training subset, wherein the programmed computer successively refines the training of the automated classifier based on successive additions, by the user using the selection interface, of further unstructured text messages to the training subset, and wherein the programmed computer evaluates the successively refined training by applying the automated classifier to an audit subset of the unstructured text messages;

the programmed computer further providing the user with a reviewable presentation of the audit subset of unstructured text messages, wherein each of the unstructured text messages of the audit subset is presented together with the category selection interface whereby the user accumulates, for each of at least a subset of the categories, a respective pool of audit instances from the audit subset for use in the evaluating; and

wherein the selection interface includes a next page control,

wherein for those unstructured text messages of the training pool for which categories have been selected on a current page, the next page control adds as training instances corresponding unstructured text together with the categories selected by the user therefor, and initiates a retraining of the automated classifier therewith, and

wherein for those unstructured text messages of the audit pool for which categories have been selected on a current page, the next page control adds as audit instances corresponding unstructured text together with the categories selected by the user therefor, and initiates classification of unstructured text messages from the newly added-to audit pool using the retrained automated classifier.

18. The system of claim 17 ,

wherein the programmed computer successively performs the evaluating of the automated classifier based on successive additions, by the user using the selection interface, of further unstructured text messages to the audit subset.

19. The system of claim 17 , further comprising:

a second programmed computer hosting a text categorization engine to which the trained, automated classifier is deployed.

20. The system of claim 17 ,

wherein the reviewable presentation of the training subset of the unstructured text messages provides a next page control that adds as training instances those unstructured text messages of a current page for which categories have been selected.

21. The system of claim 20 ,

wherein the next page control is implemented as a button.

22. The system of claim 20 , further comprising:

based on successive selections of the next page control, successively relining the training of the automated classifier based on successive additions of further unstructured text messages to the training subset.

23. A computer implemented method for identifying a set of categories for unstructured text messages and training an automated classifier therefor, the method comprising:

from a stream of the unstructured text messages captured in computer readable form, selecting a subset thereof for presentation to a user as an exploration set, the subset selected from the stream by a programmed computer;

via a display of the programmed computer, providing the user with both (i) a reviewable presentation of each unstructured text message selected for presentation as part of the exploration set and (ii) a flag definition and assignment interface, whereby the user defines categories for the unstructured text messages and flags at least one message of the exploration set as associated with each of the categories so defined;

via the display of the programmed computer, providing the user with a reviewable presentation of a training subset of the unstructured text messages, wherein each of the unstructured text messages of the training subset is presented together with a category selection interface whereby the user accumulates, for each of at least a subset of the categories, a respective pool of training instances from the training subset for use in training an automated classifier;

training an automated classifier to classify individual ones of the unstructured text messages using the training subset;

successively refining the training of the automated classifier based on successive additions, by the user using the selection interface, of further unstructured text messages to the training subset;

evaluating the successively refined training by applying the automated classifier to an audit subset of the unstructured text messages; and

via the display, providing the user with a reviewable presentation of the audit subset of unstructured text messages, wherein each of the unstructured text messages of the audit subset is presented together with the category selection interface whereby the user accumulates, for each of at least a subset of the categories, a respective pool of audit instances from the audit subset for use in the evaluating,

wherein the user interface selection includes a next page control,

wherein for those unstructured text messages of the training pool for which categories have been selected on a current page, the next page control adds as training instances corresponding unstructured text together with the categories selected by the user therefor, and initiates a retraining of the automated classifier therewith, and

wherein for those unstructured text messages of the audit pool for which categories have been selected on a current page, the next page control adds as audit instances corresponding unstructured text together with the categories selected by the user therefor, and initiates classification of unstructured text messages from the newly added-to audit pool using the retrained automated classifier.

Assignments (12)