Data compression for multidimensional time series data
Described herein are computer-implemented methods for compressing sparse multidimensional ordered series data. In particular, these methods and apparatuses for performing them (including software) may be particularly well suited to efficiently compressing spectrographic data.
1. A computer-implemented method for compressing sparse multidimensional ordered series data, the method comprising:
determining that a level of correlation of a series of similar peaks that exist between a current local region of current multidimensional ordered series data and a corresponding previous local region of a previous multidimensional ordered series data is higher or equal to a threshold, wherein the series of similar peaks are considered similar if a majority of the series of peaks in the current and previous local regions have one or more of: approximately a same mass-to-charge ratio, approximately a same charge state as determined from spacing between subsequent peaks, and similar peak intensity abundance distributions that match an avergine model;
scaling a correlated portion of the previous local region with an optimum scale factor to match a corresponding correlated portion of the current local region;
adjusting the current local region by subtracting the scaled correlated portion; and
encoding the adjusted current local region, including the optimum scale factor, into a compressed stream.
2. The method of claim 1 , wherein the current multidimensional ordered series data and the previous multidimensional ordered series data are spectrographic data.
3. The method of claim 1 , wherein the current multidimensional ordered series data and the previous multidimensional ordered series data are image data.
4. The method of claim 1 , wherein the level of correlation is determined between the current local region and an average of multiple previous local regions.
5. The method of claim 1 , wherein the previous local region corresponds to a previous local region of multiple previous local regions data that has the highest correlation with the current local region.
6. The method of claim 1 , wherein each of the current local region and the previous local region comprises one or more indexed data sets, each indexed data set comprising an index (n) and one or more variables that are indexed by the index (n).
7. The method of claim 6 , further comprising determining the current local region by dividing the current multidimensional ordered series data into a plurality of local regions, and calculating the one or more variables as a function of the index (n).
8. The method of claim 1 , wherein the previous local region corresponds an offset region of data that has already been processed.
9. The method of claim 1 , further comprising processing a plurality of local regions in an order, wherein the steps of determining that the level of correlation is higher or equal to the threshold, scaling the correlated portion of the previous local region, adjusting the current local region, and encoding the adjusted current local region are repeated for each local region in the order.
10. The method of claim 9 , wherein the order is a scan order or raster-scan order.
11. The method of claim 9 , wherein the order is selected from an order having a highest correlation level.
12. A computer-implemented method for compressing sparse multidimensional ordered series data, the method comprising:
receiving current multidimensional ordered series data, the current multidimensional ordered series data comprising spectrographic or image data;
determining that a level of correlation of a series of similar peaks that exist between a current local region of the current multidimensional ordered series data and a corresponding previous local region of a previous multidimensional ordered series data is higher or equal to a threshold, wherein the series of similar peaks are considered similar if a majority of the series of peaks in the current and previous local regions have one or more of: approximately a same mass-to-charge ratio, approximately a same charge state as determined from spacing between subsequent peaks, and similar peak intensity abundance distributions that match an avergine model;
scaling a correlated portion of the previous local region with an optimum scale factor to match a corresponding correlated portion of the current local region;
adjusting the current local region by subtracting the scaled correlated portion; and
encoding the adjusted current local region, including the optimum scale factor, into a compressed stream.
13. A system for compressing sparse multidimensional ordered series data, the system comprising a non-transitory computer-readable medium with instructions stored thereon, that when executed by a processor, cause the processor to:
determine that a level of correlation of a series of similar peaks that exist between a current local region of current multidimensional ordered series data and a corresponding previous local region of a previous multidimensional ordered series data is higher or equal to a threshold, wherein the series of similar peaks are considered similar if a majority of the series of peaks in the current and previous local regions have one or more of: approximately a same mass-to-charge ratio, approximately a same charge state as determined from spacing between subsequent peaks, and similar peak intensity abundance distributions that match an avergine model;
scale a correlated portion of the previous local region with an optimum scale factor to match a corresponding correlated portion of the current local region;
adjust the current local region by subtracting the scaled correlated portion; and
encode the adjusted current local region, including the optimum scale factor, into a compressed stream.
14. The system of claim 13 , wherein an encoder of the system encodes an identifier identifying the previous local region.
15. The system of claim 13 , wherein the level of correlation is determined between the current local region and an average of multiple previous local regions.
16. The system of claim 13 , wherein the previous local region corresponds to a previous local region of multiple previous current local regions data that has the highest correlation with the current local region.
17. The system of claim 13 , wherein each of the current local region and the previous local region comprises one or more indexed data sets, each indexed data set comprising an index (n) and one or more variables that are indexed by the index (n).
18. The system of claim 17 , further comprising determining the current local region by dividing the current multidimensional ordered series data into a plurality of local regions, and calculating the one or more variables as a function of the index (n).
19. The method of claim 1 , wherein the current and previous multidimensional ordered series data include mass spectroscopy data.
20. The system of claim 13 , wherein the current and previous multidimensional ordered series data include mass spectroscopy data.