Data tokenization and search in a distributed network
Data in a database can be protected, for instance by tokenizing the entries of the database using one or more token tables. To enable searching data within the database without first detokenizing the tokenized database entries, bigrams of each data entry can also be tokenized and stored in association with the tokenized data entry. When a query term is received, the query term can be parsed into bigrams, and each bigram can be tokenized. The tokenized query bigrams can be used to query the database, and tokenized database entries corresponding to tokenized bigrams that match the tokenized query bigrams can be identified and returned as search results.
1. A method comprising:
accessing, by a security server, a plurality of data entries in a database, each data entry comprising a set of tokenized bigrams, the set of tokenized bigrams produced by tokenizing each bigram of a set of bigrams by 1) querying a set of token tables, each token table mapping input values to token output values, with a bigram value to identify a token value mapped to the bigram value and 2) replacing the bigram value with the token value, wherein the set of bigrams is generated for a set of bytes, and wherein a first byte and a last byte of the set of bytes are included within one bigram of the set of bigrams and wherein each byte between the first byte and the last byte of the set of bytes are included within two bigrams of the set of bigrams;
receiving, by the security server, a search query term comprising a plurality of query bytes from a user;
tokenizing, by the security server, each query bigram comprising a pair of consecutive query bytes such that each query bigram includes a byte in common with at least one other query bigram using the set of token tables to produce a set of tokenized query bigrams;
querying, by the security server, the plurality of database entries with each tokenized query bigram to identify data entries associated with at least one tokenized bigram that matches at least one of the tokenized query bigrams; and
providing, by the security server, a set of tokenized bigrams associated with the identified data entries.
2. The method of claim 1 , wherein each set of bytes comprises a word of data stored within a database column.
3. The method of claim 1 , wherein each token table of the set of token tables maps each possible input value for a fixed input length to a different token value.
4. The method of claim 3 , wherein the fixed input length comprises 2 bytes.
5. The method of claim 1 , wherein the set of bytes and a corresponding set of tokenized bigrams are stored in conjunction with each other in the database.
6. The method of claim 5 , wherein the corresponding set of tokenized bigrams are stored within metadata within the database.
7. The method of claim 1 , wherein tokenizing a query bigram comprises querying a token table of the set of token tables with a value of the pair of consecutive bytes of the query bigram to identify a token value mapped to the value of the pair of consecutive bytes of the query bigram by the token table, the token value comprising the tokenized query bigram corresponding to the query bigram.
8. The method of claim 1 , wherein each identified data entry comprises a data entry associated with an above-threshold number of tokenized bigrams that match the tokenized query bigrams.
9. The method of claim 1 , wherein providing the set of tokenized bigrams comprises providing the tokenized bigrams corresponding to a threshold number of identified data entries associated with tokenized bigrams that best match the tokenized query bigrams.
10. The method of claim 1 , wherein providing the set of tokenized bigrams comprises providing all tokenized bigrams corresponding to identified data entries associated with at least a threshold number of tokenized bigrams that match the tokenized query bigrams.
11. A non-transitory computer-readable storage medium storing executable instructions that, when executed by a processor of a security server, cause the security server to perform steps comprising:
accessing, by the security server, a plurality of data entries in a database, each data entry comprising a set of tokenized bigrams, the set of tokenized bigrams produced by tokenizing each bigram of a set of bigrams by 1) querying a set of token tables, each token table mapping input values to token output values, with a bigram value to identify a token value mapped to the bigram value and 2) replacing the bigram value with the token value, wherein the set of bigrams is generated for a set of bytes, and wherein a first byte and a last byte of the set of bytes are included within one bigram of the set of bigrams and wherein each byte between the first byte and the last byte of the set of bytes are included within two bigrams of the set of bigrams;
receiving, by the security server, a search query term comprising a plurality of query bytes from a user;
tokenizing, by the security server, each query bigram comprising a pair of consecutive query bytes such that each query bigram includes a byte in common with at least one other query bigram using the set of token tables to produce a set of tokenized query bigrams;
querying, by the security server, the plurality of database entries with each tokenized query bigram to identify data entries associated with at least one tokenized bigram that matches at least one of the tokenized query bigrams; and
providing, by the security server, a set of tokenized bigrams associated with the identified data entries.
12. The non-transitory computer-readable storage medium of claim 11 , wherein each set of bytes comprises a word of data stored within a database column.
13. The non-transitory computer-readable storage medium of claim 11 , wherein each token table of the set of token tables maps each possible input value for a fixed input length to a different token value.
14. The non-transitory computer-readable storage medium of claim 13 , wherein the fixed input length comprises 2 bytes.
15. The non-transitory computer-readable storage medium of claim 11 , wherein the set of bytes and a corresponding set of tokenized bigrams are stored in conjunction with each other in the database.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the corresponding set of tokenized bigrams are stored within metadata within the database.
17. The non-transitory computer-readable storage medium of claim 11 , wherein tokenizing a query bigram comprises querying a token table of the set of token tables with a value of the pair of consecutive bytes of the query bigram to identify a token value mapped to the value of the pair of consecutive bytes of the query bigram by the token table, the token value comprising the tokenized query bigram corresponding to the query bigram.
18. The non-transitory computer-readable storage medium of claim 11 , wherein each identified data entry comprises a data entry associated with an above-threshold number of tokenized bigrams that match the tokenized query bigrams.
19. The non-transitory computer-readable storage medium of claim 11 , wherein providing the set of tokenized bigrams comprises providing the tokenized bigrams corresponding to a threshold number of identified data entries associated with tokenized bigrams that best match the tokenized query bigrams.
20. The non-transitory computer-readable storage medium of claim 11 , wherein providing the set of tokenized bigrams comprises providing all tokenized bigrams corresponding to identified data entries associated with at least a threshold number of tokenized bigrams that match the tokenized query bigrams.
21. A security server comprising:
a hardware processor; and
a non-transitory computer-readable storage medium storing executable instructions that, when executed by the hardware processor, cause the security server to perform steps comprising:
accessing, by the security server, a plurality of data entries in a database, each data entry comprising a set of tokenized bigrams, the set of tokenized bigrams produced by tokenizing each bigram of a set of bigrams by 1) querying a set of token tables, each token table mapping input values to token output values, with a bigram value to identify a token value mapped to the bigram value and 2) replacing the bigram value with the token value, wherein the set of bigrams is generated for a set of bytes, and wherein a first byte and a last byte of the set of bytes are included within one bigram of the set of bigrams and wherein each byte between the first byte and the last byte of the set of bytes are included within two bigrams of the set of bigrams;
receiving, by the security server, a search query term comprising a plurality of query bytes from a user;
tokenizing, by the security server, each query bigram comprising a pair of consecutive query bytes such that each query bigram includes a byte in common with at least one other query bigram using the set of token tables to produce a set of tokenized query bigrams;
querying, by the security server, the plurality of database entries with each tokenized query bigram to identify data entries associated with at least one tokenized bigram that matches at least one of the tokenized query bigrams; and
providing, by the security server, a set of tokenized bigrams associated with the identified data entries.