Fast compression method for scientific data
View Patent ↗A method and system allows for fast compression and decompressing of data using existing repetitive interleaved patterns within scientific data (floating point, integer, and image). An advantage of the method and system is that it is so fast that it can be used to save time due to a lower amount of data transferred/stored in scenarios like network transfer, disk or memory storage, cache storage or any other real-time applications where time plays a crucial role.
1. A method of using a computer to process data, comprising:
identifying a first starting point of the data;
initializing a stride of a number of bits;
retrieving a first group of bits at the first starting point;
jumping to a second point in the data separated by the stride from the first starting point;
retrieving a second group of bits at the second point;
comparing the first group of bits to the second group of bits; and
determining whether to update a pattern based on the comparison.
2. The method according to claim 1 , further comprising:
preparing a compressed version of the data including information regarding the pattern.
3. The method according to claim 2 , wherein the information comprises fewer bits than the pattern.
4. The method according to claim 1 , further comprising:
jumping to a third point in the data separated by the stride from the second point if there was a match between the first group of bits and the second group of bits;
retrieving a third group of bits at the third point;
comparing the first group of bits to the, third group of bits; and
determining whether to update the pattern based on the comparison.
5. The method according to claim 1 , further comprising:
resetting to a second starting point in the data if the first group of bits did not match the second group of bits;
retrieving a third group of bits at the second starting point;
jumping to a third point in the data separated by the stride from the second starting point;
retrieving a fourth group of bits at the third point;
comparing the third group of bits to the fourth group of bits; and
determining whether to update a second pattern based on the comparison.
6. The method according to claim 4 , further comprising:
resetting to a second starting point in the data if the first group of bits did not match the third group of bits, the second starting point being immediately after the first group of bits;
retrieving a third group of bits at the second starting point;
jumping to a third point in the data separated by the stride from the second starting point;
retrieving a fourth group of bits at the third point;
comparing the third group of bits to the fourth group of bits; and
determining whether to update a second pattern based on the comparison.
7. The method according to claim 1 , further comprising:
increasing the stride if the first group of bits and the second group of bits did not match;
jumping to a third point in the data separated by the increased stride from the first starting point;
retrieving a third group of bits at the third point;
comparing the first group of bits to the third group of bits; and
determining whether to update the pattern based on the comparison.
8. The method according to claim 1 , wherein the data is scientific data.
9. The method according to claim 1 , wherein the data is floating point data.
10. The method according to claim 1 , wherein the data is pixel data.
11. An apparatus for processing data, comprising:
a processor configured to initialize a value of a stride of bits and to identify a first starting point of the data and further configured to
retrieve a first group of bits at the first starting point;
jump to a second point in the data separated from the first starting point by the stride value;
retrieve a second group of bits at the second point;
compare the first group of bits to the second group of bits; and
determine whether to update a pattern based on a result of the comparison.
12. The apparatus of claim 11 , wherein the processor is further configured to prepare a compressed version of the data including information regarding the pattern.
13. The apparatus of claim 12 , wherein the information comprises fewer bits than the pattern.
14. The apparatus of claim 11 , further configured to:
jump to a third point in the data separated by the stride from the second point if there was a match between the first group of bits and the second group of bits;
retrieve a third group of bits at the third point;
compare the first group of bits to the, third group of bits; and
determine whether to update the pattern based on the comparison.