Format identification for fragmented image data
Format identification for fragmented data is disclosed. In some embodiments, an input stream of information that is divided into fragments is received. Fragments are determined and a data format for each fragment is found based on continuity properties including by: determining a physical to canonical layout transformation, determining a tuple size, and partitioning bit positions in tuples into disjoint groups wherein each group corresponds to a channel. The stream of information is compressed using a compression technique selected based on the data format, and the compressed stream is stored.
1. A system for storing information, comprising:
an interface that receives an input stream of information, wherein the input stream of information is divided into fragments;
a data model generator that determines a data format for each fragment based on continuity properties of bits comprising each fragment, wherein the data model generator:
transforms a physical layout of bits into a canonical layout of bits;
determines a tuple size in the canonical layout that results in bits in successive tuples to be correlated;
partitions bit positions in each tuple in the canonical layout into disjoint groups, wherein each group corresponds to a channel; and
identifies data format based on the determined canonical layout, tuple size, and channel partitioning;
a data compressor that compresses the input stream of information according to the identified data format of each fragment; and
a memory that stores the compressed stream.
2. The system of claim 1 , wherein the data format includes tuple size.
3. The system of claim 1 , wherein the data format includes number of channels.
4. The system of claim 1 , wherein the data format includes number of bits per channel.
5. The system of claim 1 , wherein the data format includes a position of the most significant bit.
6. The system of claim 1 , wherein for a subset of bit positions, the corresponding bits in consecutive tuples are examined and it is determined whether the joint distribution of the bits is far from being uniformly distributed.
7. The system of claim 1 , wherein the stream of information is divided into windows.
8. The method of claim 7 , wherein a size of at least one of the windows is determined based at least on part on whether the at least one window fits within a fragment of the data.
9. The system of claim 1 , wherein the data model generator determines that a fragment does not contain image data when a confidence measure does not meet a threshold.
10. The system of claim 1 , wherein bit positions within a channel are determined to be most significant based on lower variance than other bit positions.
11. The system of claim 1 , wherein the stream of information corresponds to image data.
12. The method of claim 1 , wherein the canonical layout has a repetitive structure.
13. The method of claim 12 , wherein the data format is determined based at least in part on the repetitive structure of the canonical layout.
14. The method of claim 12 , wherein the canonical layout is determined based at least in part on an endianness of a storage format of the data.
15. A method for storing information, comprising:
receiving an input stream of information, wherein the input stream of information is divided into fragments;
determining a data format for each fragment based on continuity properties of bits comprising each fragment, including by:
transforming a physical layout of bits into a canonical layout of bits;
determining a tuple size in the canonical layout that results in bits in successive tuples to be correlated;
partitioning bit positions in each tuple in the canonical layout into disjoint groups, wherein each group corresponds to a channel; and
identifying data format based on the determined canonical layout, tuple size, and channel partitioning;
compressing the input stream of information according to the identified data format of each fragment; and
storing the compressed stream.
16. The method of claim 15 , wherein the data format includes tuple size.
17. The method of claim 15 , wherein the data format includes number of channels.
18. The method of claim 15 , wherein the data format includes number of bits per channel.
19. The method of claim 15 , wherein the data format includes a position of the most significant bit.
20. The method of claim 15 , wherein for a subset of bit positions, the corresponding bits in consecutive tuples are examined and it is determined whether the joint distribution of the bits is far from being uniformly distributed.
21. The method of claim 15 , wherein the stream of information is divided into windows.
22. The method of claim 15 , wherein bit positions within a channel are determined to be most significant based on lower variance than other bit positions.
23. The method of claim 15 , wherein the stream of information corresponds to image data.
24. A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
receiving an input stream of information, wherein the input stream of information is divided into fragments;
determining a data format for each fragment based on continuity properties of bits comprising each fragment, including by:
transforming a physical layout of bits into a canonical layout of bits;
determining a tuple size in the canonical layout that results in bits in successive tuples to be correlated;
partitioning bit positions in each tuple in the canonical layout into disjoint groups, wherein each group corresponds to a channel; and
identifying data format based on the determined canonical layout, tuple size, and channel partitioning;
compressing the input stream of information according to the identified data format of each fragment; and
storing the compressed stream.