Efficient string search
View Patent ↗Some embodiments of an efficient string search have been presented. In one embodiment, a string of bytes representing content written in a non-delimited language is received, wherein the content has been classified into a predetermined category. In a single pass through the string of bytes, a set of N-grams is searched for simultaneously. Statistical information on occurrences of the N-grams, if any, in the string of bytes is collected. In some embodiments, a model is generated based on the statistical information, where the model is usable by a content filter to classify content.
1. A method for searching a string of bytes, the method comprising:
receiving a document comprising a non-delimited language over a communication network, wherein the received document has been pre-classified as being of a content type;
executing instructions stored in memory, wherein execution of the instructions by a processor:
searches the received document using a set of a plurality of N-grams,
wherein each N-gram corresponds to a pre-selected keyword for identifying content of the content type of the received document, and
wherein the search proceeds in a single pass through the received document using a finite state machine having a plurality of states, wherein the plurality of states are coupled to each other via one or more paths and the plurality of states are based on the plurality of N-grams,
determines statistical information based on occurrence of one or more N-grams found in the received document, and
generates a model for classifying as the type incoming strings of bytes representing non-segmented text written in the non-delimited language, the model generated based on the determined statistical information, wherein the model includes a predetermined number of conditions for classifying the incoming strings as the type; and
making the model available over a communication network to one or more content filters for use in classifying documents as being of the content type.
2. The method of claim 1 , further comprising generating the finite state machine, wherein generating the finite state machine comprises:
defining the plurality of states based on the plurality of N-grams;
constructing the finite state machine having the plurality of states, wherein the plurality of states are coupled to each other via one or more paths; and
mapping each of the plurality of states to the set of N-grams in an output table.
3. The method of claim 1 , further comprising collecting statistical information on occurrences of the plurality of N-grams in the string of bytes.
4. The method of claim 3 , wherein collecting the statistical information comprises counting a number of occurrences of each of the plurality of N-grams in the string of bytes.
5. The method of claim 3 , wherein searching for the plurality of N-grams in the string of bytes comprises:
inputting the string of bytes into the finite state machine; and
tracing operations of the finite state machine over the string of bytes.
6. The method of claim 5 , wherein collecting the statistical information comprises counting a number of occurrences of each of the N-grams using the output table while tracing the operations of the finite state machine.
7. The method of claim 1 , wherein the non-delimited language is selected from the group consisting of Chinese, Japanese, and Thai.
8. An apparatus for searching a string of bytes, the apparatus comprising:
a communication interface for receiving a document comprising a non-delimited language over a communication network, wherein the received document has been pre-classified as being of a content type;
a processor for executing instructions stored in memory, wherein execution of the instructions by the processor:
searches the received document using a set of a plurality of N-grams,
wherein each N-gram corresponds to a pre-selected keyword for identifying content of the content type of the received document, and
wherein the search proceeds in a single pass through the received document using a finite state machine having a plurality of states, wherein the plurality of states are coupled to each other via one or more paths and the plurality of states are based on the plurality of N-grams,
determines statistical information based on occurrence of one or more N-grams found in the received document, and
generates a model for classifying as the type incoming strings of bytes representing non-segmented text written in the non-delimited language, the model generated based on the determined statistical information, wherein the model includes a predetermined number of conditions for classifying the incoming strings as the type; and
memory for storing the model, wherein the stored model is made available over a communication network to one or more content filters for use in classifying documents as being of the content type.
9. The apparatus of claim 8 , wherein further execution of instruction by the processor generates the finite state machine, wherein the instructions for generating the finite state machine comprises instructions for:
defining the plurality of states based on the plurality of N-grams;
constructing the finite state machine having the plurality of states, wherein the plurality of states are coupled to each other via one or more paths; and
mapping each of the plurality of states to the set of N-grams in an output table.
10. The apparatus of claim 8 , wherein the memory further stores collected statistical information on occurrences of the plurality of N-grams in the string of bytes.
11. The apparatus of claim 10 , wherein the memory further stores collected statistical information on a number of occurrences of each of the plurality of N-grams in the string of bytes.
12. The apparatus of claim 10 , wherein further execution of instruction by the processor searches for the plurality of N-grams in the string of bytes, wherein the instructions for searching for the plurality of N-grams comprises instructions for:
inputting the string of bytes into the finite state machine; and
tracing operations of the finite state machine over the string of bytes.
13. The apparatus of claim 12 , wherein the memory further stores collected statistical information on a number of occurrences of each of the N-grams using the output table while tracing the operations of the finite state machine.
14. The apparatus of claim 1 , wherein the non-delimited language is selected from the group consisting of Chinese, Japanese, and Thai.
15. A non-transitory computer readable storage medium, having embodied thereon a program executable by a processor to perform a method for searching a string of bytes, the method comprising:
receiving a document comprising a non-delimited language, wherein the received document has been pre-classified as being of a content type;
searching the received document using a set of a plurality of N-grams,
wherein each N-gram corresponds to a pre-selected keyword for identifying content of the content type of the received document, and
wherein the search proceeds in a single pass through the received document using a finite state machine having a plurality of states, wherein the plurality of states are coupled to each other via one or more paths and the plurality of states are based on the plurality of N-grams;
determining statistical information based on occurrence of one or more N-grams found in the received document;
generating a model for classifying as the type incoming strings of bytes representing non-segmented text written in the non-delimited language, the model generated based on the determined statistical information, wherein the model includes a predetermined number of conditions for classifying the incoming strings as the type; and
making the model available over a communication network to one or more content filters for use in classifying documents as being of the content type.