System and method for data compression
The transmission of broadcast data, such as financial data and news feeds, is accelerated over a communication channel using data compression and decompression to provide secure transmission and transparent multiplication of communication bandwidth, as well as reduce the latency. Broadcast data may include packets having fields. Encoders associated with particular fields may be selected to compress those particular fields.
1. A method for compressing data in one or more data blocks, comprising:
analyzing a data block to determine a parameter, attribute, or value of the data block, wherein the analyzing excludes only reading a descriptor or data token associated with the data block;
selecting at least one lossless encoder associated with the determined parameter, attribute, or value;
compressing data in the data block with the selected at least one lossless encoder to produce a compressed data block, having a size over 10 times smaller than the data block; and
storing the compressed data block, wherein the time of the compressing the data block and the storing the compressed data block is less than the time of storing the data block in uncompressed form.
2. The method of claim 1 , wherein the compressing utilizes one or more state machines, and wherein at least one of the one or more state machines is a global state machine, the method further comprising selectively storing information regarding the data block in a global state machine such that the information is available to encode additional data blocks.
3. The method of claim 1 , wherein the compressing utilizes one or more state machines, and wherein at least one of the one or more state machines is a global state machine, the method further comprising selectively storing the data block in a global state machine such that the data block is available to encode additional data blocks.
4. The method of claim 1 , wherein the compressing utilizes one or more state machines, and wherein at least one of the one or more state machines is a local state machine.
5. The method of claim 1 , wherein the compressing utilizes at least two or more state machines, and wherein at least one of the two or more state machines is a global state machine, and wherein at least one of the two or more state machines is a local state machine.
6. The method of claim 1 , further comprising a plurality of data blocks, wherein the:
analyzing includes analyzing the plurality of data blocks to determine a parameter, attribute, or value of the plurality of data blocks, wherein the analyzing excludes only reading a descriptor or data token associated with the plurality of data blocks;
compressing includes compressing data in the plurality of data blocks with the selected at least one lossless encoder to produce a plurality of compressed data blocks, having a size over 10 times smaller than the plurality of data blocks; and
storing includes storing the plurality of compressed data blocks, wherein the time of the compressing the plurality of data blocks and the storing the plurality of compressed data blocks is less than the time of storing the plurality of data blocks in uncompressed form.
7. The method of claim 1 , wherein the selected at least one lossless encoder utilizes a form of mathematical analysis.
8. The method of claim 1 , wherein the selected at least one lossless encoder utilizes a dynamically loaded library.
9. A system for compressing data in one or more data blocks, comprising:
a data storage server implemented on one or more processors and one or more memory systems and configured to:
analyze a data block to determine a parameter, attribute, or value of the data block; wherein the analyzing excludes only reading a descriptor or data token associated with the data block;
select at least one lossless encoder associated with the determined parameter, attribute, or value;
compress data in the data block with the selected at least one lossless encoder to produce a compressed data block, having a size over 10 times smaller than the data block; and
store the compressed data block,
wherein the time of the compressing the data block and the storing the compressed data block is less than the time of storing the data block in uncompressed form.
10. The system of claim 9 , wherein to compress the data in the data block, the data storage server is configured to utilize one or more state machines, and wherein at least one of the one or more state machines is a global state machine, and wherein the data storage server is further configured to selectively store information regarding the data block in a global state machine such that the information is available to encode additional data blocks.
11. The system of claim 9 , wherein to compress the data in the data block, the data storage server is configured to utilize one or more state machines, and wherein at least one of the one or more state machines is a global state machine, and wherein the data storage server is further configured to selectively store the data block in a global state machine such that the data block is available to encode additional data blocks.
12. The system of claim 9 , wherein to compress the data in the data block, the data storage server is further configured to utilize one or more state machines, and wherein at least one of the one or more state machines is a local state machine.
13. The system of claim 9 , wherein to compress the data in the data block, the data storage server is further configured to utilize at least two or more state machines, and wherein at least one of the two or more state machines is a global state machine, and wherein at least one of the two or more state machines is a local state machine.
14. The system of claim 9 , further comprising a plurality of data blocks, wherein the data storage server is further configured to:
analyze the plurality of data blocks to determine a parameter, attribute, or value of the plurality of data blocks, wherein the analyzing excludes only reading a descriptor or data token associated with the plurality of data blocks;
compress data in the plurality of data blocks with the selected at least one lossless encoder to produce a plurality of compressed data blocks, having a size over 10 times smaller than the plurality of data blocks; and
store the plurality of compressed data blocks, wherein the time of the compressing the plurality of data blocks and the storing the plurality of compressed data blocks is less than the time of storing the plurality of data blocks in uncompressed form.
15. The system of claim 9 , wherein the selected at least one lossless encoder utilizes a form of mathematical analysis.
16. The system of claim 9 , wherein the selected at least one lossless encoder utilizes a library.
17. A method for compressing data in one or more data blocks, comprising:
analyzing a data block to determine a parameter, attribute, or value of the data block; wherein the analyzing excludes only reading a descriptor or data token associated with the data block;
selecting at least one lossless compression encoder associated with the determined parameter, attribute, or value;
compressing data in the data block with the selected at least one lossless compression encoder to produce a compressed data block, having a size over 10 times smaller than the data block;
transmitting the compressed data block; and
storing the compressed data block, wherein the time of the compressing, the transmitting, and the storing of the data block is less than the time of transmitting and storing the data block in uncompressed form.
18. The method of claim 17 , wherein transmitting the compressed data block further comprises transmitting one or more data packets to a client, wherein the one or more data packets includes control information and compressed data information.
19. The method of claim 17 , wherein the data block is transmitted in one or more data packets utilizing Transmission Control Protocol/Internet Protocol (TCP/IP), the one or more data packets including control information and compressed data information.
20. The method of claim 17 , wherein transmitting the compressed data block further comprises:
sequencing the compressed data block with one or more additional compressed data blocks to produce a sequence of aggregated data blocks; and
transmitting the sequence of aggregated data blocks.
21. The method of claim 17 , further comprising:
after transmitting the compressed data block and before storing the compressed data block, decompressing the compressed data block, wherein the time to compress, transmit, decompress, and store is less than the time of transmitting and storing the data block in uncompressed form.
22. The method of claim 17 , wherein the compressing utilizes one or more state machines, and wherein at least one of the one or more state machines is a global state machine, the method further comprising selectively storing information regarding the data block in a global state machine such that the information is available to encode additional data blocks.
23. The method of claim 17 , wherein the compressing utilizes one or more state machines, and wherein at least one of the one or more state machines is a global state machine, the method further comprising selectively storing the data block in a global state machine such that the data block is available to encode additional data blocks.
24. The method of claim 17 , wherein the compressing utilizes one or more state machines, and wherein at least one of the one or more state machines is a local state machine.
25. The method of claim 17 , wherein the compressing utilizes at least two or more state machines, and wherein at least one of the two or more state machines is a global state machine, and wherein at least one of the two or more state machines is a local state machine.
26. The method of claim 17 , further comprising a plurality of data blocks, wherein the:
analyzing includes analyzing the plurality of data blocks to determine a parameter, attribute, or value of the plurality of data blocks, wherein the analyzing excludes only reading a descriptor or data token associated with the plurality of data blocks;
compressing includes compressing data in the plurality of data blocks with the selected at least one lossless encoder to produce a plurality of compressed data blocks, having a size over 10 times smaller than the plurality of data blocks; and
storing includes storing the plurality of compressed data blocks, wherein the time of the compressing the plurality of data blocks and the storing the plurality of compressed data blocks is less than the time of storing the plurality of data blocks in uncompressed form.