Format identification for fragmented image data
Format identification for fragmented data is disclosed. In some embodiments, an input stream of information that is divided into fragments is received. Fragment boundaries are determined and a data format for each fragment is found based on continuity properties including by: dividing the stream of information into windows, determining whether each window has a known or unknown format; and comparing portions of windows having an unknown format with neighboring windows to determine fragment boundaries. The stream of information is compressed using a compression technique selected based on the data format, and the compressed stream is stored.
1. A system for storing information, comprising:
an interface that receives an input stream of information, wherein the input stream of information comprises a plurality of fragments;
a data model generator that is configured to determine fragment boundaries of the plurality of fragments and to determine a data format for each of the plurality of fragments based on continuity properties;
wherein the data model generator operates to
divide the input stream of information into a plurality of windows, wherein each window has a fixed size and includes a same number of bytes, and wherein each of the plurality of fragments contains no more than a single window;
for each of the plurality of windows:
determine whether the window has a known or unknown format based on a value of a scoring function, wherein a high value of the scoring function indicates the window has a known format whereas a low value of the scoring function indicates the window has an unknown format;
compare portions of the window having an unknown format with neighboring windows to determine fragment boundaries;
calculate statistics of bits in the window based on formats of the neighboring windows; and
identify a breakpoint based on the statistics of bits in the window;
a data compressor that compresses the plurality of fragments into a compressed stream using a compression technique selected based on the data format for each of the fragments; and
a memory that stores the compressed stream.
2. The system of claim 1 , wherein the data model generator compares portions of the window having an unknown format with neighboring windows by checking whether the window matches the format of a previous window.
3. The system of claim 2 , wherein the data model generator checks whether the window matches the format of a previous window by checking for correlations between successive tuples in the window.
4. The system of claim 2 , wherein the data model generator checks whether the window matches the format of a previous window by checking for correlations between successive tuples in the window and shifting an alignment within the window.
5. The system of claim 1 , wherein each window comprises a plurality of tuples.
6. The system of claim 1 , wherein a window having an unknown format is assigned the format of its neighboring windows when the neighboring windows have the same format.
7. The system of claim 1 , wherein a break point is determined within a window having an unknown format and having neighbors with different formats.
8. The system of claim 1 , wherein the stream of information corresponds to image data.
9. A method for storing information, comprising:
receiving, at an interface, an input stream of information, wherein the input stream of information comprises a plurality of fragments;
determining, by a data model generator, fragment boundaries of the plurality of fragment and determining a data format for each of the plurality of fragments based on continuity properties by performing the steps comprising:
dividing the input stream of information into a plurality of windows, wherein each window has a fixed size and includes a same number of bytes, and wherein each of the plurality of fragments contains no more than a single window;
for each of the plurality of windows:
determining whether each window has a known or unknown format based on a value of a scoring function, wherein a high value of the scoring function indicates the window has a known format whereas a low value of the scoring function indicates the window has an unknown format;
comparing portions of the window having an unknown format with neighboring windows to determine fragment boundaries;
calculating statistics of bits in the window based on formats of the neighboring windows; and
identifying a breakpoint based on the statistics of bits in the window;
compressing the plurality of fragments into a compressed stream using a compression technique selected based on the data format for each of the fragments; and
storing the compressed stream.
10. The method of claim 9 , wherein comparing portions of the window having an unknown format with neighboring windows comprises checking whether the window matches the format of a previous window.
11. The method of claim 10 , wherein checking whether the window matches the format of a previous window comprises checking for correlations between successive tuples in the window.
12. The method of claim 10 , wherein checking whether the window matches the format of a previous window comprises checking for correlations between successive tuples in the window and shifting an alignment within the window.
13. The method of claim 9 , wherein each window comprises a plurality of tuples.
14. The method of claim 9 , wherein a window having an unknown format is assigned the format of its neighboring windows when the neighboring windows have the same format.
15. The method of claim 9 , wherein a break point is determined within a window having an unknown format and having neighbors with different formats.
16. The method of claim 9 , wherein the stream of information corresponds to image data.
17. A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
receiving, at an interface, an input stream of information, wherein the input stream of information comprises a plurality of fragments;
determining, by a data model generator, fragment boundaries of the plurality of fragments and determining a data format for each of the plurality of fragments based on continuity properties by performing the steps comprising:
dividing the input stream of information into a plurality of windows, wherein each window has a fixed size and includes a same number of bytes, and wherein each of the plurality of fragments contains no more than a single window;
for each of the plurality of windows:
determining whether each window has a known or unknown format based on a value of a scoring function, wherein a high value of the scoring function indicates the window has a known format whereas a low value of the scoring function indicates the window has an unknown format;
comparing portions of the window having an unknown format with neighboring windows to determine fragment boundaries;
calculating statistics of bits in the window based on formats of the neighboring windows; and
identifying a breakpoint based on the statistics of bits in the window;
compressing the plurality of fragments into a compressed stream using a compression technique selected based on the data format for each of the fragments; and
storing the compressed stream.
18. The computer program product of claim 17 , wherein the stream of information corresponds to image data.
19. The computer program product of claim 17 , wherein the data model generator compares portions of the window having an unknown format with neighboring windows by checking whether the window matches the format of a previous window.
20. The computer program product of claim 19 , wherein the data model generator checks whether the window matches the format of a previous window by checking for correlations between successive tuples in the window and shifting an alignment within the window.