Systems and methods for tokenizing user-generated content to enable the prevention of attacks
The present invention relates to systems and methods for the tokenization of user-generated content in order to prevent attacks on the user-generated content. The systems and methods initially pre-process the user-generated content string utilizing a secondary input of target language. Pre-processing may also include initialization of finite state machines, token markers and string buffers (text, HTML tag name, HTML attribute name, HTML attribute value, CSS selector, CSS property name, and CSS property value). The user-generated content string is scanned by rune, and the system sends each rune to a specific buffer based upon signaling by individual finite state machine states. Buffers are then converted to token stream nodes to be inserted into the token stream. The tokens represent a string of characters and are symbolically categorized according to activated finite state machine states.
1. A method for tokenizing user-generated content, using a computer implemented security system, the method comprising:
pre-processing a user-generated content input string utilizing a secondary input of target language, wherein the pre-preprocessing converts existing text into a token stream text node at the start of an HTML tag for insertion into a token stream; and
extracting tokens, using a processor, from the pre-processed user-generated content string to generate the token stream, wherein the token stream is yielded to a caller rather than the user-generated content to prevent attacks on the user-generated content;
wherein the extraction of tokens from the pre-processed user-generated content requires scanning the pre-processed user-generated content string by individual runes, and sending each rune to a specific buffer based upon signaling individual finite state machines.
2. The method of claim 1 , wherein the tokens represent a string of characters or equivalent in bytes.
3. The method of claim 2 , wherein the tokens are categorized according to rules as a symbol.
4. The method of claim 3 , wherein the extraction is performed through an activation of finite state machines.
5. The method of claim 4 , wherein the tokens belong to one of a plurality of buffers, wherein the plurality of buffers include at least one of text, HTML tag name, HTML attribute name, HTML attribute value, CSS selector, CSS property name, and CSS property value.
6. The method of claim 5 , further comprising a conversion of buffers into nodes and subsequently appending those nodes to the token stream.
7. The method of claim 6 , further comprising inserting remaining buffers that did not activate finite state machine states into the token stream as token stream nodes.
8. The method of claim 7 , further comprising returning the token stream, comprised of token stream nodes.
9. The method of claim 5 , further comprising clearing finite state machine variables, token markers, and string buffers at the end of the tokenizing.
10. A security system for tokenizing user-generated content comprising:
a pre-processor configured to process a user-generated content input string utilizing a secondary input of target language, wherein the pre-processor converts existing text into a token stream text node at the start of an HTML tag for insertion into a token stream; and
a tokenizer, including a processor, configured to extract tokens from the pre-processed user-generated content string to generate the token stream, wherein the token stream is yielded to a caller rather than the user-generated content to prevent attacks on the user-generated content;
wherein the tokenizer scans the pre-processed user-generated content by individual runes, and sends each rune to a specific buffer based upon signaling individual finite state machine states.
11. The system of claim 10 , wherein the tokens represent a string of characters or equivalent in bytes.
12. The system of claim 11 , wherein the tokens are categorized according to rules as a symbol.
13. The system of claim 12 , wherein the extraction of tokens is performed through activation of finite state machines.
14. The system of claim 13 , wherein the tokens belong to one of a plurality of buffers, wherein the plurality of buffers include at least one of text, HTML tag name, HTML attribute name, HTML attribute value, CSS selector, CSS property name, and CSS property value.
15. The system of claim 14 , wherein the tokenizer is further configured to convert buffers into strings which are HTML encoded, and appending the strings to the token stream as token stream nodes.
16. The system of claim 15 , wherein the tokenizer is further configured to insert remaining buffers that did not activate finite state machines into the token stream as token stream nodes.
17. The system of claim 16 , wherein the tokenizer is further configured to return the token stream full of token stream nodes.
18. The system of claim 14 , wherein the tokenizer is further configured to clear finite state machine variables, tokenizer markers, and string buffers at the end of tokenizing.