Relational method for transforming unsorted sparse dictionary encodings into unsorted-dense or sorted-dense dictionary encodings
Unsorted sparse dictionary encodings are transformed into unsorted-dense or sorted-dense dictionary encodings. Sparse domain codes have large gaps between codes that are adjacent in order. Unlike spare codes, dense codes have smaller gaps between adjacent codes; consecutive codes are dense codes that have no gaps between adjacent codes. The techniques described herein are relational approaches that may be used to generate sparse composite codes and sorted codes.
1. A method for encoding a base table, the method comprising:
executing a first database statement that specifies to create an unsorted-encoded table from a base table that contains a token column wherein executing said first database statement causes generating said unsorted-encoded table, wherein said token column contains a plurality of tokens that includes a plurality of distinct tokens, wherein said unsorted-encoded table maps a plurality of unsorted codes to said plurality of distinct tokens, wherein said plurality of unsorted codes do not reflect an order of said plurality of distinct tokens, wherein each unsorted code of said plurality of unsorted codes is mapped by said unsorted-encoded table to a respective distinct token of said plurality of distinct tokens;
executing a second database statement that specifies to create an unsorted-to-sorted table that maps said plurality of unsorted codes to a plurality of sorted codes, wherein executing said second database statement causes generating said unsorted-to-sorted table, wherein each unsorted code of said plurality of unsorted codes is mapped by said unsorted-to-sorted table to a sorted code of said plurality of sorted codes that reflects the order of the respective distinct token of said each unsorted code among the plurality of distinct tokens;
based on the unsorted-to-sorted table, generating a dictionary table, wherein each sorted code of said plurality of sorted codes is mapped by said dictionary table to a respective distinct token of said plurality of distinct tokens; and
encoding the token column based on the dictionary table.
2. The method of claim 1 , wherein the first database statement applies a first encode operator to said token column, wherein said first encode operator generates a distinct code for each distinct token.
3. The method of claim 2 , wherein the first database statement applies a second encode operator to said token column, wherein said second encode operator stores, in a flag column of said unsorted-encode table, flags that each indicate whether a respective unsorted code is a first occurrence of said respective unsorted code.
4. The method of claim 3 , the method further including executing a particular database statement that causes:
removing duplicates by filtering on said flag column; and
for each row of said unsorted-to-sorted table, generate a sorted code to map to an unsorted code in said each row by ranking a token in said token column of said each row against tokens in said token column.
5. The method of claim 4 , wherein said particular database statement includes a window ranking function that ranks over said token column based on an order of tokens in said token column.
6. One or more non-transitory computer-readable media storing one or more sequences instructions for encoding a base table, wherein said one or more sequences of instructions, when executed by one or more processors, cause:
executing a first database statement that specifies to create an unsorted-encoded table from a base table that contains a token column, wherein executing said first database statement causes generating said unsorted-encoded table, wherein said token column contains a plurality of tokens that includes a plurality of distinct tokens, wherein said unsorted-encoded table maps a plurality of unsorted codes to said plurality of distinct tokens, wherein said plurality of unsorted codes do not reflect an order of said plurality of distinct tokens, wherein each unsorted code of said plurality of unsorted codes is mapped by said unsorted-encoded table to a respective distinct token of said plurality of distinct tokens;
executing a second database statement that specifies to create an unsorted-to-sorted table that maps said plurality of unsorted codes to a plurality of sorted codes, wherein executing said second database statement causes generating said unsorted-to-sorted table, wherein each unsorted code of said plurality of unsorted codes is mapped by said unsorted-to-sorted table to a sorted code of said plurality of sorted codes that reflects the order of the respective distinct token of said each unsorted code among the plurality of distinct tokens;
based on the unsorted-to-sorted table, generating a dictionary table, wherein each sorted code of said plurality of sorted codes is mapped by said dictionary table to a respective distinct token of said plurality of distinct tokens; and
encoding the token column based on the dictionary table.
7. The one or more non-transitory computer-readable media of claim 6 , wherein the first database statement applies a first encode operator to said token column, wherein said first encode operator generates a distinct code for each distinct token.
8. The one or more non-transitory computer-readable media of claim 7 , wherein the first database statement applies a second encode operator to said token column, wherein said second encode operator stores, in a flag column of said unsorted-encode table, flags that each indicate whether a respective unsorted code is a first occurrence of said respective unsorted code.
9. The one or more non-transitory computer-readable media of claim 8 , wherein the one or more sequences of instructions include instructions, that when executed by said one or more processors, cause generating the unsorted-to-sorted table includes executing a particular database statement that causes:
removing duplicates by filtering on said flag column; and
for each row of said unsorted-to-sorted table, generate a sorted code to map to an unsorted code in said each row by ranking a token in said token column of said each row against tokens in said token column.
10. The one or more non-transitory computer-readable media of claim 9 , wherein said particular database statement includes a window ranking function that ranks over said token column based on an order of tokens in said token column.