Method and system for accelerated morphological analysis
View Patent ↗A system provides accelerated morphological analysis and in particular a speed-up of morphological look-up via a caching mechanism. The system determines whether each incoming token in a token stream is unique or recurring. Unique tokens, which occur for the first time in the token stream, are marked with a unique numerical identification (ID). A pointer is added to recurring tokens, which already occurred in the token stream, and directed towards the unique numerical ID which was defined for the respective token when occurring for the first time. A morphological look-up is performed on the unique tokens. Subsequently, the tokens carrying the pointer are detected and replaced with the results of morphological look-up stored under the unique numerical ID of the respective unique token.
1. A method for performing morphology analysis of a natural language document, comprising the steps of:
inputting the natural language document as an input text,
tokenize the input text, thereby producing a token stream of a plurality of tokens,
comparing each token of the token stream to a list of previously encountered tokens in the token stream,
determining whether the token either corresponds to at least one previously encountered token in the list as a recurring token in the token stream, or else is a unique token in the token stream without corresponding to the at least one previously encountered token in the list,
adding the unique token to the list of previously encountered tokens, marking the unique token with an identification,
adding a pointer directed towards the identification for the recurring token,
performing a morphological look-up only on the unique token, thereby producing a morphological look-up result,
storing said morphological look-up result for the unique token together with the identification,
reading the morphological look-up result for the recurring token,
joining the morphological look-up results for the unique and recurring tokens, thereby producing a stream of morphological analyses, and
outputting the stream of morphological analyses.
2. The method according to claim 1 , wherein the step of storing the morphological look-up result for the unique token together with the identification comprises the step of creating a dynamically extending database.
3. The method according to claim 2 , wherein said dynamically extending database is a self-extending hash table.
4. The method according to claim 1 , wherein the step of tokenizing is performed by a first finite state transducer.
5. The method according to claim 4 , wherein the first finite state transducer includes punctuation conventions and higher level lexical information.
6. The method according to claim 4 , wherein the step of morphological look-up is performed by a second finite state transducer.
7. The method according to claim 1 , wherein the step of joining the morphological look-up results further comprises replacing the each token and the recurring token with the pointer from the morphological look-up unless the each token is marked with the pointer.
8. A system for performing morphology analysis of a natural language document, comprising:
a tokenizer for tokenizing an input document, thereby producing a token stream of a plurality of tokens,
a pre-processor that compares each token of the token stream to a list of previously encountered tokens in the token stream, determines whether the token either corresponds to at least one previously encountered token in the list as a recurring token in the token stream, or else is a unique token in the token stream without corresponding to the at least one previously encountered token in the list, adds the unique token to the list of previously encountered tokens, marks the unique token with an identification and adds a pointer directed towards the identification for the recurring token,
a morphological look-up module for performing a morphological look-up only on the unique token, thereby producing a morphological look-up result,
memory for storing said morphological look-up result for the unique token together with the identification,
a post-processor that replaces a token that carries the pointer with the morphological look-up result stored in the memory under the respective identification.
9. The system according to claim 8 , wherein the tokenizer is a first finite state transducer.
10. The system according to claim 9 , wherein the tokenizer includes punctuation conventions and higher level lexical information.
11. The system according to claim 9 , wherein the morphological look-up module is a second finite state transducer.
12. The system according to claim 8 , wherein the memory is a dynamically extending database.
13. The system according to claim 12 , wherein said dynamically extending database is a self-extending hash table.
14. The system according to claim 8 , wherein memory stores mapping between the unique token and the identification.
15. A system for performing morphology analysis of a natural language document, comprising:
a tokenizer for tokenizing an input document, thereby producing a token stream of a plurality of tokens,
a morphological look-up module for producing results of a morphological look-up,
memory for storing the results of the morphological look-up for a unique token of the plurality of tokens together with an identification,
a control unit for controlling the morphological look-up module and the memory; wherein the control unit:
compares each token of the token stream to a list of previously encountered tokens in the token stream, determines whether the token either corresponds to at least one previously encountered token in the list as a recurring token in the token stream, or else is the unique token in the token stream without corresponding to the at least one previously encountered token in the list,
adds the unique token to the list of previously encountered tokens,
marks the unique token with the identification,
adds a pointer directed towards the identification for the recurring token,
initializes the morphological look-up for the unique token, and
replaces the recurring token marked with the pointer by the results of the morphological look-up stored in the memory under the respective identification.
16. The system according to claim 15 , wherein the tokenizer is a first finite state transducer.
17. The system according to claim 16 , wherein the tokenizer includes punctuation conventions and higher level lexical information.
18. The system according to claim 16 , wherein the morphological look-up module is a second finite state transducer.
19. The system according to claim 15 , wherein the memory is a dynamically extending database.
20. The system according to claim 19 , wherein said dynamically extending database is a self-extending hash table.