Method and system for identifying sentence boundaries
The present invention is directed to systems and methods for isolating sentence boundaries between sentences in text. Sentences of the normalized document feeds or source text are separated by determining boundaries between individual sentences, by a Bayesian algorithm, that has been seeded with rule frequencies, developed from a previous training phase, that employed a text of sentences with marked boundaries between the sentences.
1 . A method for separating individual sentences in text, comprising:
using a Bayesian algorithm to analyze the text; and,
classify boundaries of sentences within the text.
2 . The method of claim 1 , additionally comprising:
providing a training text with its sentences separated by indicators to seed the Bayesian algorithm.
3 . The method of claim 2 , wherein the boundaries are classified as sentence or non-sentence boundaries.
4 . A method for separating individual sentences in text, comprising:
providing at least two rules to classify portions of the text as a sentence or a non-sentence;
tagging a training text with sentence boundaries; and,
applying a Bayesian analysis to the training text to determine thresholds for marking sentence boundaries, whereby a program is populated with rules.
5 . The method of claim 4 , wherein the populated program is applied to the text to locate sentences.
6 . The method of claim 4 , wherein the precision of the sentence boundaries is measured by the percentage of valid positives, false positives and missed positives.