Data compression utilizing variable and limited length codes
View Patent ↗According to one embodiment, method for decoding encoded data comprises a hardware module including circuitry to process a data stream. The data stream includes one or more encoded symbols each including a code assigned to a corresponding symbol. A set of least frequently occurring symbols are assigned a common code to encode those symbols within the data stream. Data blocks are generated each containing a data stream portion. One or more encoded symbols within each data block are identified by comparing data block portions to maximum code values for corresponding code lengths to determine a code length for each data block portion. A starting location for the identified encoded symbols within each data block is determined based on the determined code lengths.
1. A method of decoding encoded data comprising:
generating data blocks each containing a portion of a data stream, wherein the data stream includes one or more encoded symbols each including a code assigned to a corresponding symbol, and wherein a set of least frequently occurring symbols are assigned a common code to encode those symbols within the data stream;
identifying one or more encoded symbols within each data block by comparing portions of that data block to maximum code values for corresponding code lengths to determine a code length for each portion of that data block; and
determining a starting location for the one or more identified encoded symbols within each data block based on the determined code lengths.
2. The method of claim 1 , wherein the symbols are encoded by a Huffman based coding scheme.
3. The method of claim 1 , wherein the encoded symbols are identified by a Field Programmable Gate Array (FPGA) within one clock cycle.
4. The method of claim 1 , wherein the encoded symbols within the data stream represent one of a column and a value of a database table.
5. The method of claim 1 , wherein the codes assigned to the symbols are limited to a predetermined maximum code length.
6. The method of claim 1 , wherein the common code is followed in the data stream by a literal value of a corresponding one of the least frequently occurring symbols.
7. The method of claim 1 , wherein determining a starting location for the one or more identified encoded symbols includes:
determining a starting location for a group of two or more successive encoded symbols within the data block based on the determined code lengths.
8. The method of claim 1 , wherein the symbols include one or more trailing space characters that are collectively assigned a code omitted from the data stream.
9. The method of claim 1 , further including:
retrieving codes for the identified one or more encoded symbols from each data block based on the corresponding starting locations and determining the corresponding symbols for the one or more identified encoded symbols based on the retrieved codes.
10. An apparatus for decoding encoded data comprising:
a hardware module including circuitry to process a data stream, wherein the data stream includes one or more encoded symbols each including a code assigned to a corresponding symbol, and wherein a set of least frequently occurring symbols are assigned a common code to encode those symbols within the data stream, the hardware module configured to:
generate data blocks each containing a portion of the data stream;
identify one or more encoded symbols within each data block by comparing portions of that data block to maximum code values for corresponding code lengths to determine a code length for each portion of that data block; and
determine a starting location for the one or more identified encoded symbols within each data block based on the determined code lengths.
11. The apparatus of claim 10 , wherein the symbols are encoded by a Huffman based coding scheme.
12. The apparatus of claim 10 , wherein the circuitry includes a Field Programmable Gate Array (FPGA), and the encoded symbols are identified within one clock cycle.
13. The apparatus of claim 10 , wherein the encoded symbols within the data stream represent one of a column and a value of a database table.
14. The apparatus of claim 10 , wherein the codes assigned to the symbols are limited to a predetermined maximum code length.
15. The apparatus of claim 10 , wherein the common code is followed in the data stream by a literal value of a corresponding one of the least frequently occurring symbols.
16. The apparatus of claim 10 , wherein determining a starting location for the one or more identified encoded symbols includes:
determining a starting location for a group of two or more successive encoded symbols within the data block based on the determined code lengths.
17. The apparatus of claim 10 , wherein the symbols include one or more trailing space characters that are collectively assigned a code omitted from the data stream.
18. The apparatus of claim 10 , wherein the hardware module is further configured to:
retrieve codes for the identified one or more encoded symbols from each data block based on the corresponding starting locations and determine the corresponding symbols for the one or more identified encoded symbols based on the retrieved codes.
19. The apparatus of claim 10 , wherein the circuitry includes a processor.
20. A computer program product for decoding encoded data comprising:
a non-transitory computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising computer readable program code configured to:
generate data blocks each containing a portion of a data stream, wherein the data stream includes one or more encoded symbols each including a code assigned to a corresponding symbol, and wherein a set of least frequently occurring symbols are assigned a common code to encode those symbols within the data stream;
identify one or more encoded symbols within each data block by retrieving code information for portions of that data block from a memory, wherein the memory includes an entry including the code information for each value of a data portion and the entry is retrievable based on the value of the data portion; and
determine a code length and starting location for the one or more identified encoded symbols within each data block based on the retrieved code information.
21. The computer program product of claim 20 , wherein the symbols are encoded by a Huffman based coding scheme.
22. The computer program product of claim 20 , wherein the encoded symbols within the data stream represent one of a column and a value of a database table.
23. The computer program product of claim 20 , wherein the codes assigned to the symbols are limited to a predetermined maximum code length.
24. The computer program product of claim 20 , wherein the memory indicates for the common code a literal value of a corresponding one of the least frequently occurring symbols.
25. The computer program product of claim 20 , wherein determining a code length and starting location for the one or more identified encoded symbols includes:
determining code lengths and a starting location for a group of two or more successive encoded symbols within the data block based on the retrieved code information.
26. The computer program product of claim 20 , wherein the symbols include one or more trailing space characters that are collectively assigned a code omitted from the data stream.
27. The computer program product of claim 20 , wherein the computer readable program code further includes computer readable program code configured to:
retrieve codes for the identified one or more encoded symbols from each data block based on the corresponding starting locations and determine the corresponding symbols for the one or more identified encoded symbols based on the retrieved codes.