Content-aware compression of data using window-based selection from multiple prediction functions

1. A method for compressing at least one floating point number, comprising the steps of:

obtaining said at least one floating point number represented using one or more bits to indicate a sign of said at least one floating point number, one or more bits to indicate an exponent at a given base and one or more bits to indicate a significand of said at least one floating point number, wherein said significand has a length equal to a number of bits between a most significant bit of said significand and a least significant bit of said significand having a predefined binary value;

applying a predefined default prediction algorithm and a plurality of other distinct prediction algorithms to said obtained at least one floating point number to generate a corresponding plurality of predictions;

evaluating a compression metric over a collection of floating point numbers, including said at least one floating point number, for said predefined default prediction algorithm and said plurality of other distinct prediction algorithms; and

based on said compression metric, encoding one of:

said at least one floating point number by encoding, as a single code, the exponent and the length of a residual generated by said predefined default prediction algorithm; or

said collection of floating point numbers by encoding, as single codes, the exponent and the lengths of the residuals generated by an alternate prediction algorithm from among said plurality of other distinct prediction algorithms.

2. The method of claim 1 , wherein said compression metric indicates, for said at least one floating point number and a given prediction algorithm, said number of bits saved if said given prediction algorithm is used to encode said collection of floating point numbers.

3. The method of claim 2 , wherein said compression metric is used to select a given prediction algorithm that substantially maximizes bit savings across said collection of floating point numbers.

4. The method of claim 3 , wherein two substantially local maxima of said substantially maximized bit savings must be a predefined number of samples apart.

5. The method of claim 1 , further comprising the step of storing an indication of whether said predefined default prediction algorithm is used for said encoding of said at least one floating point number or said alternate prediction algorithm is used for said encoding of said collection of floating point numbers.

6. The method of claim 1 , further comprising the step of storing an index of said alternate prediction algorithm among a plurality of available alternate prediction algorithms.

7. The method of claim 6 , further comprising the step of storing an indication of said plurality of available alternate prediction algorithms as metadata.

8. The method of claim 1 , further comprising the steps of decompressing said encoded at least one floating point number by evaluating whether said predefined default prediction algorithm or said alternate prediction algorithm was used for said encoding and evaluating an index of said alternate prediction function among a plurality of available alternate prediction algorithms if said alternate prediction function was used for said encoding.

9. The method of claim 1 , further comprising the step of determining a set of one or more prediction algorithms out of a larger set of prediction algorithms for a specific data set including said at least one floating point number based on a performance-based ranking of the prediction algorithms of the larger set of prediction algorithms with respect to the specific data set including said at least one floating point number, so that the total number of bits saved over all floating point numbers is substantially maximal.

10. The method of claim 1 , wherein said predefined default prediction algorithm and a plurality of available alternate prediction algorithms correspond to ensembles of prediction algorithms, and wherein the method further comprises the steps of:

storing an ensemble index to specify an ensemble of prediction algorithms when there is more than one possible ensemble to be selected; and

storing an indication of said ensembles of prediction algorithms.

11. The method of claim 10 , further comprising the step of storing a disambiguation index associated with said encoding of said at least one floating point number, or with said encoding of each element of said collection of floating point numbers, indicating one selected prediction algorithm from a potential subset of said ensemble of prediction algorithms.

12. The method of claim 11 , wherein said potential subset is generated by discarding the predictions that, when added to said residual, correspond to a floating point number for which there would be a better prediction resulting in another residual that can be represented with fewer bits.

13. The method of claim 12 , wherein a set of default prediction algorithms and said plurality of available alternate prediction algorithms are selected for a segment of said data set including said at least one floating point number, wherein said data set is segmented based on one or more of a local variance, a local average, a local measure of smoothness and a local auto-correlation, said method optionally comprising the step of storing said disambiguation index associated with said encoding of said at least one floating point number, or with said encoding of each element of said collection of floating point numbers, indicating one selected prediction algorithm from said potential subset of said ensemble of prediction algorithms.

14. The method of claim 13 , further comprising the steps of decompressing said encoded at least one floating point number by evaluating whether a default ensemble of prediction algorithms or an alternate ensemble of prediction algorithms was used for said encoding and evaluating an index of said ensemble of prediction algorithms and evaluating said disambiguation index of said prediction algorithm used for said encoding.

15. A computer program product for compressing at least one floating point number, comprising a non-transitory machine-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed by at least one processing device perform the following steps:

obtaining said at least one floating point number represented using one or more bits to indicate a sign of said at least one floating point number, one or more bits to indicate an exponent at a given base and one or more bits to indicate a significand of said at least one floating point number, wherein said significand has a length equal to a number of bits between a most significant bit of said significand and a least significant bit of said significand having a predefined binary value;

applying a predefined default prediction algorithm and a plurality of other distinct prediction algorithms to said obtained at least one floating point number to generate a corresponding plurality of predictions;

evaluating a compression metric over a collection of floating point numbers, including said at least one floating point number, for said predefined default prediction algorithm and said plurality of other distinct prediction algorithms; and

based on said compression metric, encoding one of:

said at least one floating point number by encoding, as a single code, the exponent and the length of a residual generated by said predefined default prediction algorithm; or

said collection of floating point numbers by encoding, as single codes, the exponent and the lengths of the residuals generated by an alternate prediction algorithm from among said plurality of other distinct prediction algorithms.

16. The computer program product of claim 15 , further comprising one or more of the steps of storing an indication of whether said predefined default prediction algorithm is used for said encoding of said at least one floating point number or said alternate prediction algorithm is used for said encoding of said collection of floating point numbers and storing an index of said alternate prediction algorithm among a plurality of available alternate prediction algorithms.

17. The computer program product of claim 15 , further comprising the steps of decompressing said encoded at least one floating point number by evaluating whether said predefined default prediction algorithm or said alternate prediction algorithm was used for said encoding and evaluating an index of said alternate prediction function among a plurality of available alternate prediction algorithms if said alternate prediction function was used for said encoding.

18. The computer program product of claim 15 , wherein said predefined default prediction algorithm and a plurality of available alternate prediction algorithms correspond to ensembles of prediction algorithms, and wherein the computer program product further comprises the steps of:

storing an ensemble index to specify an ensemble of prediction algorithms when there is more than one possible ensemble to be selected; and

storing an indication of said ensembles of prediction algorithms.

19. A system for compressing at least one floating point number, comprising:

a memory; and

at least one processing device, coupled to the memory, operative to implement the following steps:

obtaining said at least one floating point number represented using one or more bits to indicate a sign of said at least one floating point number, one or more bits to indicate an exponent at a given base and one or more bits to indicate a significand of said at least one floating point number, wherein said significand has a length equal to a number of bits between a most significant bit of said significand and a least significant bit of said significand having a predefined binary value;

applying a predefined default prediction algorithm and a plurality of other distinct prediction algorithms to said obtained at least one floating point number to generate a corresponding plurality of predictions;

evaluating a compression metric over a collection of floating point numbers, including said at least one floating point number, for said predefined default prediction algorithm and said plurality of other distinct prediction algorithms; and

based on said compression metric, encoding one of:

said at least one floating point number by encoding, as a single code, the exponent and the length of a residual generated by said predefined default prediction algorithm; or

said collection of floating point numbers by encoding, as single codes, the exponent and the lengths of the residuals generated by an alternate prediction algorithm from among said plurality of other distinct prediction algorithms.

20. The system of claim 19 , wherein said compression metric is used to select a given prediction algorithm that substantially maximizes bit savings across said collection of floating point numbers and wherein two substantially local maxima of said substantially maximized bit savings must be a predefined number of samples apart.

21. The system of claim 19 , further comprising the step of storing an indication of whether said predefined default prediction algorithm is used for said encoding of said at least one floating point number or said alternate prediction algorithm is used for said encoding of said collection of floating point numbers.

22. The system of claim 19 , further comprising the step of storing an index of said alternate prediction algorithm among a plurality of available alternate prediction algorithms.

23. The system of claim 19 , further comprising the steps of decompressing said encoded at least one floating point number by evaluating whether said predefined default prediction algorithm or said alternate prediction algorithm was used for said encoding and evaluating an index of said alternate prediction function among a plurality of available alternate prediction algorithms if said alternate prediction function was used for said encoding.

24. The system of claim 19 , further comprising the step of determining a set of one or more prediction algorithms out of a larger set of prediction algorithms for a specific data set including said at least one floating point number based on a performance-based ranking of the prediction algorithms of the larger set of prediction algorithms with respect to the specific data set including said at least one floating point number, so that the total number of bits saved over all floating point numbers is substantially maximal.

25. The system of claim 19 , wherein said predefined default prediction algorithm and a plurality of available alternate prediction algorithms can correspond to ensembles of prediction algorithms, and wherein the system further comprises the steps of:

storing an ensemble index to specify an ensemble of prediction algorithms when there is more than one possible ensemble to be selected; and

storing an indication of said ensembles of prediction algorithms.