Apparatus and method for text extraction

1. A method of determining main text in a mark-up document, comprising:

removing, by a system having a processor, first predetermined mark-up tags from the mark-up document, and replacing second predetermined mark-up tags in the mark-up document with separation elements, wherein the removing and the replacing cause the mark-up document to contain text paragraphs and the separation elements without the first and second predetermined mark-up tags;

determining, by the system, a length of each of the text paragraphs in the mark-up document; and

determining, by the system, one or more main paragraphs of the mark-up document based upon the lengths of the text paragraphs in the mark-up document.

2. The method of claim 1 , further comprising:

normalizing the lengths of the text paragraphs in the mark-up document with respect to a length of a longest text paragraph in the mark-up document, wherein determining the one or more main paragraphs is based on the normalized lengths of the text paragraphs.

3. The method of claim 2 , wherein the lengths of the text paragraphs are normalized according to the equation:

n

i

′

=

n

i

n

max

wherein n′ i is a normalized length of a text paragraph, n i is a length of a text paragraph, 1≦i≦N is a number of a last text paragraph in the mark-up document, and n max is the length of the longest text paragraph in the mark-up document.

4. The method of claim 1 , comprising:

smoothing the lengths of the text paragraphs in the mark-up document based upon one or more adjacent text paragraph lengths, wherein determining the one or more main paragraphs is based on the smoothed lengths.

5. The method of claim 4 , wherein the lengths of the text paragraphs are smoothed according to the equation:

n

i

″

=

{

w

×

n

1

′

+

n

2

′

w

+

1

,

i

=

1

n

i

-

1

′

+

w

×

n

i

′

+

n

i

+

1

′

1

+

w

+

1

,

1

<

i

<

N

n

N

-

1

′

+

w

×

n

N

′

1

+

w

,

i

=

N

wherein n″ i is a smoothed length of a text paragraph, n i is a length of a text paragraph, and w is a predetermined smoothing weight.

6. The method of claim 1 , wherein the replacing comprises replacing the second predetermined mark-up tags in the mark-up document with line breaks.

7. The method of claim 1 , wherein the replacing comprises replacing the second predetermined mark-up tags with carriage returns.

8. The method of claim 1 , comprising:

determining first main paragraphs of the mark-up document based upon the lengths of the text paragraphs in the mark-up document;

discarding from consideration the first main paragraphs of the mark-up document and determining second main paragraphs of the mark-up document based upon the lengths of the text paragraphs remaining in the mark-up document.

9. The method of claim 8 , further comprising combining the first and second main paragraphs as the main paragraphs of the mark-up document.

10. An apparatus for determining main text in a mark-up document, comprising:

a memory having a mark-up document stored therein; and

a processor configured to:

remove first predetermined mark-up tags from the mark-up document, and replace second predetermined mark-up tags in the mark-up document with separation elements, wherein the removing and the replacing cause the mark-up document to contain text paragraphs and the separation elements without the first and second predetermined mark-up tags;

determine lengths of the text paragraphs in the mark-up document; and

determine one or more main paragraphs of the mark-up document based upon the lengths of the text paragraphs in the mark-up document.

11. The apparatus of claim 10 , wherein the processor is configured to further compute a paragraph length threshold based upon the lengths of the text paragraphs in the mark-up document, and wherein the determining of the one or more main paragraphs is further according to the paragraph length threshold.

12. The apparatus of claim 10 , wherein the processor is configured to further normalize the lengths of the text paragraphs in the mark-up document with respect to a length of a longest text paragraph in the mark-up document, wherein the determining of the one or more main paragraphs is based on the normalized lengths of the text paragraphs.

13. A non-transitory computer-readable storage medium storing instructions that upon execution cause a system to:

remove first predetermined mark-up tags from the mark-up document, and replacing second predetermined mark-up tags in the mark-up document with separation elements, wherein the removing and the replacing cause the mark-up document to contain text paragraphs and the separation elements without the first and second predetermined mark-up tags;

determine a length of each of the text paragraphs in the mark-up document; and

determine one or more main paragraphs of the mark-up document based upon the lengths of the text paragraphs in the mark-up document.

14. The non-transitory computer-readable storage medium of claim 13 , wherein the instructions upon execution cause the system to further:

normalize the lengths of the text paragraphs in the mark-up document with respect to a length of a longest text paragraph in the mark-up document, wherein determining the one or more main paragraphs is based on the normalized lengths of the text paragraphs.

15. The non-transitory computer-readable storage medium of claim 14 , wherein the lengths of the text paragraphs are normalized according to the equation:

n

i

′

=

n

i

n

max

wherein n′ i is a normalized length of a text paragraph, n i is a length of a text paragraph, 1≦i≦N is a number of a last text paragraph in the mark-up document, and n max is the length of the longest text paragraph in the mark-up document.

16. The non-transitory computer-readable storage medium of claim 13 , wherein the instructions upon execution cause the system to further:

smooth the lengths of the text paragraphs in the mark-up document based upon one or more adjacent text paragraph lengths, wherein determining the one or more main paragraphs is based on the smoothed lengths.

17. The non-transitory computer-readable storage medium of claim 13 , wherein the replacing comprises replacing the second predetermined mark-up tags in the mark-up document with line breaks.

18. The non-transitory computer-readable storage medium of claim 13 , wherein the replacing comprises replacing the second predetermined mark-up tags with carriage returns.

19. The non-transitory computer-readable storage medium of claim 13 , wherein the instructions upon execution cause the system to further:

determine first main paragraphs of the mark-up document based upon the lengths of the text paragraphs in the mark-up document;

discard from consideration the first main paragraphs of the mark-up document and determining second main paragraphs of the mark-up document based upon the lengths of the text paragraphs remaining in the mark-up document.

20. The non-transitory computer-readable storage medium of claim 19 , wherein the instructions upon execution cause the system to further combine the first and second main paragraphs as the main paragraphs of the mark-up document.

21. The non-transitory computer-readable storage medium of claim 13 , wherein the instructions upon execution cause the system to further compute a paragraph length threshold based upon the lengths of the text paragraphs in the mark-up document, and wherein the determining of the one or more main paragraphs is further according to the paragraph length threshold.