Efficient string search
Some embodiments of an efficient string search have been presented. In one embodiment, a string of bytes representing content written in a non-delimited language is received, wherein the content has been classified into a predetermined category. In a single pass through the string of bytes, a set of N-grams is searched for simultaneously. Statistical information on occurrences of the N-grams, if any, in the string of bytes is collected. In some embodiments, a model is generated based on the statistical information, where the model is usable by a content filter to classify content.
1. A method for classifying content written in a non-delimited language, the method comprising:
receiving a string of bytes at a finite state machine (FSM), wherein the string of bytes is received by electronic hardware associated with the FSM after a user attempts to access information related to the string of bytes;
performing a string search on the string of bytes, wherein the string search identifies that the string of bytes includes a set of N-grams that match one or more states in a set of states, wherein the FSM connects the one or more states in the set of states;
collecting statistical information regarding the set of N-grams received in the string of bytes, wherein the collected statistical information corresponds to a condition in a model stored in a model repository;
receiving the model from the model repository;
identifying that the one or more states and that one or more N-grams in the set of N-grams correspond to the received model;
identifying that a length of the received string of bytes also corresponds to the condition in the received model when the length of the received string of bytes is of a certain length;
classifying content of the one or more N-grams as being prohibited according to the one or more states, the one or more N-grams, and the length that corresponds to the condition; and
denying access to the content when the classification of the content is prohibited, wherein denying access to the content prevents the content from being displayed on a display accessible to the user after the user attempt to access the information relating to the string of bytes.
2. The method of claim 1 , further comprising, counting the number of N-grams that correspond to the one or more states in the string of bytes when collecting the statistical information.
3. The method of claim 2 , wherein another model is generated when the number of N-grams counted is equal to or greater than a predetermined number.
4. The method of claim 2 , wherein:
the content is classified when the number of N-grams counted is equal to or greater than a predetermined number, and
the content classified is written in a non-delimited language.
5. The method of claim 1 , wherein the model is received by a content filter.
6. The method of claim 1 , wherein the one or more N-grams of the set of N-grams are stored when a finite state machine transitions to a state that is mapped to the one or more N-grams.
7. A non-transitory computer readable storage medium having embodied thereon a program executable by a processor to perform a method for classifying content written in a non-delimited language, the method comprising:
receiving a string of bytes by the processor, wherein the processor executing instructions out of the memory performs the function of a finite state machine (FSM) after a user attempts to access information related to the string of bytes;
performing a string search on the string of bytes, wherein the string search determines that the string of bytes matches one or more states in the set of states;
performing a string search on the string of bytes, wherein the string search identifies that the string of bytes includes a set of N-grams that match one or more states in a set of states, wherein the FSM connects the one or more states in the set of states;
collecting statistical information regarding the set of N-grams received in the string of bytes, wherein the collected statistical information corresponds to a condition in a model stored in a model repository;
receiving the model from the model repository;
identifying that the one or more states and that one or more N-grams in the set of N-grams correspond to the received model;
identifying that a length of the received string of bytes also corresponds to the condition in the received model when the length of the received string of bytes is of a certain length;
classifying content of the one or more N-grams as being prohibited according to the one or more states, the one or more N-grams, and the length that corresponds to the condition; and
denying access to the content when the classification of the content is prohibited, wherein denying access to the content prevents the content from being displayed on a display accessible to the user after the user attempt to access the information relating to the string of bytes.
8. The non-transitory computer readable storage medium of claim 7 , the program further executable to count the number of N-grams that correspond to the one or more states in the string of bytes when collecting the statistical information.
9. The non-transitory computer readable storage medium of claim 8 , wherein another model is generated when the number of N-grams counted is equal to or greater than a predetermined number.
10. The non-transitory computer readable storage medium of claim 8 , wherein:
the content is classified when the number of N-grams counted is equal to or greater than a predetermined number, and
the content classified is written in a non-delimited language.
11. The non-transitory computer readable storage medium of claim 7 , wherein the model is received by a content filter.
12. The non-transitory computer readable storage medium of claim 7 , wherein the one or more N-grams of the set of N-grams are stored when a finite state machine transitions to a state that is mapped to the one or more N-grams.
13. A system for classifying content written in a non-delimited language, the system comprising:
a processor; and
a memory, wherein the processor:
receives a string of bytes after a user attempts to access information related to the string of bytes;
performs a string search on the string of bytes according to a finite state machine (FSM), wherein the string search identifies that the string of bytes includes a set of N-grams that match one or more states in a set of states;
collects statistical information regarding the set of N-grams received in the string of bytes, wherein the collected statistical information corresponds to a condition in a model stored in a model repository;
receives the model from the model repository;
identifies that the one or more states and that one or more N-grams in the set of N-grams corresponds to the received model;
identifies that a length of the received string of bytes also corresponds to the condition in the received model when the length of the received string of bytes is of a certain length;
classify content of the one or more N-grams as being prohibited according to the one or more states, the one or more N-grams, and the length that corresponds to the condition; and
denies access to the content when the classification of the content is prohibited, wherein denying access to the content prevents the content from being displayed on a display accessible to the user after the user attempt to access the information relating to the string of bytes.
14. The system of claim 13 , wherein the processor counts the number of N-grams that correspond to the one or more states in the string of bytes when collecting the statistical information.
15. The system of claim 14 , wherein another model is generated when the number of N-grams counted is equal to or greater than a predetermined number.
16. The system of claim 14 , wherein the processor:
classifies the content when the number of N-grams counted is equal to or greater than a predetermined number; and
the content classified is written in a non-delimited language.
17. The system of claim 13 , further comprising a network, wherein a request is received over the network when the user attempts to access the information related to the string of bytes.
18. The system of claim 13 , wherein the processor stores in the memory the one or more N-grams of the set of N-grams when a finite state machine transitions to a state that is mapped to the one or more N-grams.