Dynamic dictionary-based data symbol encoding
View Patent ↗A dynamic dictionary-based data symbol encoder. A dynamic dictionary data structure is populated with evictable dictionary entries. The evictable dictionary entries are encoded with a dictionary index that is shorter than an original representation of the input symbols. A reference count evicts dictionary indices when eligible for eviction. Through building a dynamic symbol dictionary which is much smaller than (global) alphabet size, locally repetitive symbols can be effectively compressed using dictionary. The dictionary is also dynamically built along with the compression/decompression process and therefore does not carry overhead. However, tables/trees might be appended to enable entropy decoding. The method is also readily combined with the popular LZ77 and its variant encoding methods into composite one-pass encoding algorithms to achieve superior performance.
1. An encoding method comprising:
populating a dynamic dictionary data structure with a plurality of evictable dictionary entries based on input symbols of an input data stream, said evictable dictionary entries including first and second evictable dictionary entries;
encoding said evictable dictionary entries with a dictionary index that is shorter than an original representation of the of the input symbols;
maintaining a dynamic mapping table from the input symbols to dictionary index;
associating, with each one of said evictable dictionary entries, a reference count that is initialized to a predetermined minimum threshold wherein said reference count is increasable from the predetermined minimum threshold to the predetermined maximum threshold; and
using a cyclic pointer that rotates among said evictable dictionary entries to determine whether to replace a currently pointed dictionary entry,
wherein the reference count for the first evictable dictionary entry is reduced by one if: (1) an input symbol in the input data stream is distinct from all dictionary entries, (2) the cyclic pointer is at the first evictable dictionary entry, and (3) said reference count for the first evictable dictionary entry is greater than the predetermined minimum threshold.
2. The encoding method of claim 1 wherein the first evictable dictionary entry is eligible for eviction when its reference count is reduced to the predetermined minimum threshold.
3. The encoding method of claim 1 wherein the first evictable dictionary entry, upon eviction, is replaced by an input symbol from the input data stream that is distinct from the first and second evictable dictionary entries.
4. The encoding method of claim 1 further comprising outputting an encoded data stream wherein the encoded data stream includes an indicator bit followed by an output data symbol.
5. The encoding method of claim 4 wherein the indicator bit specifies that the output data symbol is unencoded.
6. The encoding method of claim 4 wherein the indicator bit specifies that the output data symbol is encoded.
7. The encoding method of claim 4 wherein the encoded data stream includes a first indicator bit and a second indicator bit wherein the first indicator bit specifies that the output data symbol is unencoded and the second indicator bit specifies that the output data symbol is encoded.
8. The encoding method of claim 1 wherein the dictionary size is a power of 2.
9. The encoding method of claim 1 wherein the dictionary is a 5-bit dictionary.
10. A non-transitory machine-readable storage media having instructions for performing an encoding method which causes a computing device to:
register, in a dynamic dictionary data structure, a plurality of evictable dictionary entries that are input symbols from an input data stream with said evictable dictionary entries including first and second evictable dictionary entries;
encode said evictable dictionary entries with a dictionary index that is shorter than an original representation of the input symbols;
associate, with each one of said evictable dictionary entries, a reference count that is initialized to a predetermined minimum threshold wherein said reference count is increasable from the predetermined minimum threshold to the predetermined maximum threshold; and
provide a cyclic pointer that rotates among said evictable dictionary entries to determine whether to replace a currently pointed dictionary entry,
wherein the reference count for the first evictable dictionary entry is reduced by one if: (1) an input symbol in the input data stream is different from all dictionary entries, (2) the cyclic pointer is at the first evictable dictionary entry, and (3) the reference count for the first evictable dictionary entry is greater than the predetermined minimum threshold.
11. The non-transitory machine-readable storage media of claim 10 wherein said predetermined minimum threshold is zero and said predetermined maximum threshold is three.
12. The non-transitory machine-readable storage media of claim 10 wherein said reference count for the first evictable dictionary entry is increased by one whenever the first evictable dictionary entry is encoded with its dictionary index.
13. The non-transitory machine-readable storage media of claim 10 wherein said reference count for said first evictable dictionary is increased by one until the predetermined maximum threshold is reached.