Block compression algorithm
View Patent ↗A method for compressing a data stream based on a combination of two digests is used. Each combination of digests that occurs multiple times is assigned a codeword. When the first digest matches, a data structure of second digests occurring with the same first digest is built, associating each combination of digests with a position in the stream where the combination of digests is found. When a new match is found a new codeword is defined. When a match with an existing codeword is found, a byte sequence is emitted that identifies the codeword associated with the matched data. When a partial match is found, the length of a codeword is changed and a corresponding byte sequence is emitted. The method finds particular application in data streams that are sent to printers, and which contain large blocks of identical data such as images or repetitive text.
1. A computer-implemented method for encoding a data stream of bytes, said method comprising the steps of:
(a) receiving said data stream;
(b) providing a fixed-length lookahead window for said data stream, the lookahead window having a start position;
(c) determining a first digest of said window;
(d) determining if the first digest of said window has a corresponding entry in a primary data structure;
(e) storing in the primary data structure, a position in the data stream corresponding to the start position of said window if the primary data structure does not include an entry corresponding to the first digest of said window;
(f) if the primary data structure includes an entry corresponding to the first digest of said window, modifying the entry to include a secondary data structure of second digests each occurring with the same first digest and associating each second digest included in the secondary data structure with a respective position in the data stream where the combination of the first and second digests was found;
(g) if the primary data structure already contains a secondary data structure of second digests, determining if the second digest of the current lookahead window is present in the secondary data structure contained in the primary data structure;
(h) if the second digest of the current lookahead window is not present in the secondary data structure contained in the primary data structure, modifying the secondary data structure contained in the primary data structure to associate the second digest of the current lookahead window with the start position of the lookahead window in the data stream;
(i) if the second digest of the current lookahead window is present in the secondary data structure contained in the primary data structure, determining a length of a matching portion by reading first data from a first location in the data stream at which the combination of first and second digests was found and comparing the first data from the first location with second data starting at the start position of the current lookahead window, and storing the length of the matching portion in the secondary data structure and assigning a codeword to the sequence of bytes in the matching portion;
(j) if the combination of digests for the current lookahead window already has an associated length and the length of the matching portion is greater than zero and less than the associated length, the length of a codeword assigned to the combination of digests of the current lookahead window is shortened in accordance with the length of the matching portion;
(k) continuing processing at a location in the data stream of a next non-matching byte until the data stream has been fully processed;
(l) after completing steps (a)-(j), further processing blocks of unmatched data by a dictionary coder adapted to find small repetitions of data;
wherein the method is performed by a computer programmed to perform steps (a)-(l).
2. The method according to claim 1 , wherein said data stream is a print stream.
3. The method according to claim 1 , wherein the primary data structure is a lookup table and secondary data structure is a binary tree.
4. The method according to claim 1 , wherein each digests has a length of three bytes, the first digest consisting of a first byte, a middle byte and a second-to-last byte of the lookahead window and the second digest consisting of a second byte, a byte after the middle byte and a last byte of the lookahead window.
5. The method according to claim 1 , wherein the first and second digests are created by splitting a single digest into two parts.
6. The method according to claim 1 , wherein the combination of first and second digests are stored at intervals of bytes of the data stream, the intervals being of no more than half the size of the lookahead window.
7. The method according to claim 1 , wherein at step (i), the length of the matching portion is determined by further reading the data stream backward from the start position of the lookahead window to find a position at which a longer match can be found.
8. A computer-implemented method of generating a compressed data stream with the method for encoding in accordance with claim 1 , wherein the compressed data stream comprises blocks of data interspersed with codeword definitions and usage sequences.
9. The method according to claim 8 further comprising identifying lengths of other blocks of data with length bytes.
10. The method according to claim 8 , further comprising processing blocks of unmatched data of said data stream of bytes by a dictionary coder adapted to find small repetitions of data.
11. A computer-implemented method for encoding a data stream of bytes, said method comprising the steps of:
(a) receiving said data stream;
(b) providing a fixed-length lookahead window for said data stream, the lookahead window having a start position;
(c) determining a first digest of said window;
(d) determining if the first digest of said window has a corresponding entry in a primary data structure;
(e) storing in the primary data structure, a position in the data stream corresponding to the start position of said window if the primary data structure does not include an entry corresponding to the first digest of said window;
(f) if the primary data structure includes an entry corresponding to the first digest of said window, modifying the entry to include a secondary data structure of second digests each occurring with the same first digest and associating each second digest included in the secondary data structure with a respective position in the data stream where the combination of the first and second digests was found;
(g) if the primary data structure already contains a secondary data structure of second digests, determining if the second digest of the current lookahead window is present in the secondary data structure contained in the primary data structure;
(h) if the second digest of the current lookahead window is not present in the secondary data structure contained in the primary data structure, modifying the secondary data structure contained in the primary data structure to associate the second digest of the current lookahead window with the start position of the lookahead window in the data stream;
(i) if the second digest of the current lookahead window is present in the secondary data structure contained in the primary data structure, determining a length of a matching portion by reading first data from a first location in the data stream at which the combination of first and second digests was found and comparing the first data from the first location with second data starting at the start position of the current lookahead window, and storing the length of the matching portion in the secondary data structure and assigning a codeword to the sequence of bytes in the matching portion;
(j) if the combination of digests for the current lookahead window already has an associated length and the length of the matching portion is greater than zero and less than the associated length, the length of a codeword assigned to the combination of digests of the current lookahead window is shortened in accordance with the length of the matching portion; and
(k) continuing processing at a location in the data stream of a next non-matching byte until the data stream has been fully processed;
wherein the method is performed by a computer programmed to perform steps (a)-(k).