Text segmentation with multiple granularity levels

1. A method of text processing, comprising:

training, using a processor, a classifier for classifying text, wherein:

the training is based on a plurality of training sample entries;

a training sample entry in the plurality of training sample entries includes:

a character count;

an independent use rate;

a phrase structure rule value indicating whether the training sample entry complies with phrase structure rules;

a semantic attribute value indicating an inclusion state of the training sample entry in a predetermined set of enumerated entries;

an overlap attribute value indicating overlap of the training sample entry with another entry in the predetermined set of enumerated entries; and

a classification result indicating whether the training sample entry is a compound semantic unit or a smallest semantic unit;

building, using the processor, a lexicon of smallest semantic units, comprising:

receiving an entry to be classified;

using the trained classifier to determine whether the entry to be classified is a smallest semantic unit or a compound semantic unit; and

in the event that the entry is determined to be a smallest semantic unit, adding the entry to the lexicon of smallest semantic units;

segmenting, using the processor, received text based on the lexicon of smallest semantic units to obtain medium-grained segmentation results;

merging, using the processor, the medium-grained segmentation results to obtain coarse-grained segmentation results, the coarse-grained segmentation results having coarser granularity than the medium-grained segmentation results;

looking up, using the processor, in the lexicon of smallest semantic units respective search elements that correspond to segments in the medium-grained segmentation results; and

forming, using the processor, fine-grained segmentation results based on the respective search elements, the fine-grained segmentation results having finer granularity than the medium-grained segmentation results.

2. The method of claim 1 , wherein the received text is in a non-divider marked language.

3. The method of claim 1 further comprising, in the event that the entry is determined to be a compound semantic unit, adding the entry to a lexicon of compound semantic units.

4. The method of claim 1 , wherein using the trained classifier to determine whether the entry is a smallest semantic unit or a compound semantic unit includes inputting into the trained classifier: a character count of the entry, an independent use rate of the entry, a phrase structure rule indicator indicating whether the entry complies with phrase structure rules, a semantic attribute indicating an inclusion state of the entry in the predetermined set of enumerated entries, and an overlap attribute of the entry.

5. The method of claim 1 , further comprising:

determining a search element that corresponds to the entry; and

saving the search element in the lexicon of smallest semantic units.

6. The method of claim 5 , wherein determining a search element that corresponds to the entry includes:

determining whether the entry is segmentable;

in the event that the entry is segmentable, setting the search element to a fine-grained word included within the entry; and

in the event that the entry is not segmentable, setting the search element to the entry.

7. The method of claim 1 , wherein segmenting received text based on the lexicon of smallest semantic units to obtain medium-grained segmentation results include resolving an ambiguity of the medium-grained segmentation results.

8. A system for text processing, comprising:

one or more processors configured to:

train a classifier for classifying text, wherein:

the training is based on a plurality of training sample entries;

a training sample entry in the plurality of training sample entries includes:

a character count;

an independent use rate;

a phrase structure rule value indicating whether the training sample entry complies with phrase structure rules;

a semantic attribute value indicating an inclusion state of the training sample entry in a predetermined set of enumerated entries;

an overlap attribute value indicating overlap of the training sample entry with another entry in the predetermined set of enumerated entries; and

a classification result indicating whether the training sample entry is a compound semantic unit or a smallest semantic unit; and

build a lexicon of smallest semantic units, comprising:

receiving an entry to be classified;

using the trained classifier to determine whether the entry to be classified is a smallest semantic unit or a compound semantic unit; and

in the event that the entry is determined to be a smallest semantic unit, adding the entry to the lexicon of smallest semantic units;

segment received text based on the lexicon of smallest semantic units to obtain medium-grained segmentation results;

merge the medium-grained segmentation results to obtain coarse-grained segmentation results, the coarse-grained segmentation results having coarser granularity than the medium-grained segmentation results;

look up in the lexicon of smallest semantic units respective search elements that correspond to segments in the medium-grained segmentation results; and

form fine-grained segmentation results based on the respective search elements, the fine-grained segmentation results having finer granularity than the medium-grained segmentation results; and

one or more memories coupled to the one or more processors, configured to provide the one or more processors with instructions.

9. The system of claim 8 , wherein the text is in a non-divider marked language.

10. The system of claim 8 , wherein the one or more processors are further configured to, in the event that the entry is determined to be a compound semantic unit, add the entry to a lexicon of compound semantic units.

11. The system of claim 8 , wherein using the trained classifier to determine whether the entry is a smallest semantic unit or a compound semantic unit includes inputting into the trained classifier: a character count of the entry, an independent use rate of the entry, a phrase structure rule indicator indicating whether the entry complies with phrase structure rules, a semantic attribute indicating an inclusion state of the entry in the predetermined set of enumerated entries, and an overlap attribute of the entry.

12. The system of claim 8 , wherein the one or more processors are further configured to:

determine a search element that corresponds to the entry; and

save the search element in the lexicon of smallest semantic units.

13. The system of claim 12 , wherein determining a search element that corresponds to the entry includes:

determining whether the entry is segmentable;

in the event that the entry is segmentable, setting the search element to a fine-grained word included within the entry; and

in the event that the entry is not segmentable, setting the search element to the entry.

14. The system of claim 8 , wherein segmenting received text based on the lexicon of smallest semantic units to obtain medium-grained segmentation results includes resolving an ambiguity of the medium-grained segmentation results.

15. A computer program product for text processing, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

training a classifier for classifying text, wherein:

the training is based on a plurality of training sample entries;

a training sample entry in the plurality of training sample entries includes:

a character count;

an independent use rate;

a phrase structure rule value indicating whether the training sample entry complies with phrase structure rules;

a semantic attribute value indicating an inclusion state of the training sample entry in a predetermined set of enumerated entries;

an overlap attribute value indicating overlap of the training sample entry with another entry in the predetermined set of enumerated entries; and

a classification result indicating whether the training sample entry is a compound semantic unit or a smallest semantic unit; and

building a lexicon of smallest semantic units, comprising:

receiving an entry to be classified;

using the trained classifier to determine whether the entry to be classified is a smallest semantic unit or a compound semantic unit; and

in the event that the entry is determined to be a smallest semantic unit, adding the entry to a lexicon of smallest semantic units;

segmenting received text based on the lexicon of smallest semantic units to obtain medium-grained segmentation results;

merging the medium-grained segmentation results to obtain coarse-grained segmentation results, the coarse-grained segmentation results having coarser granularity than the medium-grained segmentation results;

looking up in the lexicon of smallest semantic units respective search elements that correspond to segments in the medium-grained segmentation results; and

forming fine-grained segmentation results based on the respective search elements, the fine-grained segmentation results having finer granularity than the medium-grained segmentation results.

16. A system for text processing, comprising:

one or more processors configured to:

train a classifier for classifying text, wherein:

the training is based on a plurality of training sample entries;

a training sample entry in the plurality of training sample entries includes:

a character count;

an independent use rate;

a phrase structure rule value indicating whether the training sample entry complies with phrase structure rules;

a semantic attribute value indicating an inclusion state of the training sample entry in a predetermined set of enumerated entries;

an overlap attribute value indicating overlap of the training sample entry with another entry in the predetermined set of enumerated entries; and

a classification result indicating whether the training sample entry is a compound semantic unit or a smallest semantic unit; and

build a lexicon of smallest semantic units, comprising:

receiving an entry to be classified;

using the trained classifier to determine whether the entry to be classified is a smallest semantic unit or a compound semantic unit; and

in the event that the entry is determined to be a smallest semantic unit, adding the entry to the lexicon of smallest semantic units; and

one or more memories coupled to the one or more processors, configured to provide the one or more processors with instructions.