Methods and systems for compressing data

1. A system for compressing data to reduce index size, the system comprising:

a processor; and

a machine-readable medium in operable communication with the processor and comprising instructions stored thereon that, when executed by the processor, perform the following steps:

receiving a fragment-ion index comprising the data, the data comprising mass spectrometry data;

transforming the data to reduce its entropy and give transformed data; and

encoding the transformed data to reduce an amount of memory used by the transformed data and to give a custom data structure with reduced size compared to the received index,

the custom data structure being searchable without requiring any decompression prior to searching, such that both individual fragment-ions of the custom data structure and mass spectrometry data of the custom data structure are searchable without requiring any decompression prior to searching,

the mass spectrometry data comprising mass spectrometry data of one or more peptides, and

the transforming of the data comprising:

i) representing each theoretical mass spectrometry spectrum of the data as an ordered list of numbers where each number represents the m/z (mass divided by charge) of a predicted fragment-ion in the spectrum;

ii) ordering ions in each generated spectrum S i first by their ion-series (b/y), and then by their fragment charge, and then by their m/z values;

iii) grouping peptide sequences first by their length and then using all theoretical spectra in each group, where X represents the quantity of theoretical spectra in each group;

iv) constructing a matrix N=[S 1 , S 2 , . . . , S X ] as

N

=

[

i

11

¨

i

1

⁢

k

¨

i

1

⁢

n

i

j

⁢

⁢

1

¨

i

jk

¨

i

jn

i

X

⁢

⁢

1

¨

i

Xk

¨

i

Xn

]

,

where i jk corresponds to the m/z of k th fragment-ion in the j th spectrum, 0<i jk ≤(M−1), and M is maximum fragment-ion mass;

v) re-enumerating the ions to form a flattened list T=[i 0 , i 1 , i 2 , . . . , i Xn-1 ];

vi) defining a list A containing indices of ions in T, where A=[0, 1, 2, . . . , Xn −1 ], where a dimension of N is given as dim(N)=(X×n); and

vii) performing a Stable KeyValue Sort operation on A using T as a key to yield the transformed data in array Ω and array Λ (Ω, Λ=StableKeyValueSort(T,A)).

2. The system according to claim 1 , the encoding of the transformed data comprising:

viii) transforming Ω into a list Ω′ by storing only a number of occurrences of each appearing unique ion-mass (Ω′[i]={occs [i]; i€[0, M−1]}), where |Ω′|=M<<X n =|Ω|.

3. The system according to claim 2 , the encoding of the transformed data further comprising:

ix) computing an equivalent list Ω″ given as Ω″[i]=Ω″[i−1]+Ω′[i−1] by successively accumulating occurrences in Ω′, where Ω″ corresponds to start positions in A for each indexed unique ion-mass.

4. The system according to claim 2 , the transforming and encoding further comprising performing steps v)-viii) on all instances of N to construct the complete fragment-ion index.

5. The system according to claim 1 , the transforming of the data further comprising performing steps v)-vii) on all instances of N.

6. The system according to claim 1 , the encoding of the transformed data comprising using a running counter technique to compress repetitive and redundant information in the transformed data.

7. The system according to claim 1 , the encoding of the transformed data comprising using a running counter technique to compress repetitive and redundant information in the transformed data.

8. A method for compressing data to reduce index size, the method comprising:

receiving a fragment-ion index comprising the data, the data comprising mass spectrometry data;

transforming, by a processor, the data to reduce its entropy and give transformed data, and

encoding, by the processor, the transformed data to reduce an amount of memory used by the transformed data and to give a custom data structure with reduced size compared to the received index,

the custom data structure being searchable without requiring any decompression prior to searching, such that both individual fragment-ions of the custom data structure and mass spectrometry data of the custom data structure are searchable without requiring any decompression prior to searching,

the mass spectrometry data comprising mass spectrometry data of one or more peptides, and

the transforming of the data comprising:

i) representing each theoretical mass spectrometry spectrum of the data as an ordered list of numbers where each number represents the m/z (mass divided by charge) of a predicted fragment-ion in the spectrum;

ii) ordering ions in each generated spectrum S; first by their ion-series (b/y), and then by their fragment charge, and then by their m/z values;

iii) grouping peptide sequences first by their length and then using all theoretical spectra in each group, where X represents the quantity of theoretical spectra in each group;

iv) constructing a matrix N=[S 1 , S 2 , . . . , S X ] as

N

=

[

i

11

¨

i

1

⁢

k

¨

i

1

⁢

n

i

j

⁢

⁢

1

¨

i

jk

¨

i

jn

i

X

⁢

⁢

1

¨

i

Xk

¨

i

Xn

]

,

where i jk corresponds to the m/z of k th fragment-ion in the j th spectrum, 0<i jk ≤(M−1), and M is maximum fragment-ion mass;

v) re-enumerating the ions to form a flattened list T=[i 0 , i 1 , i 2 , . . . , i Xn-1 ];

vi) defining a list A containing indices of ions in T, where A=[0, 1, 2, . . . , Xn −1 ], where a dimension of N is given as dim(N)=(X×n); and

vii) performing a Stable KeyValue Sort operation on A using T as a key to yield the transformed data in array Ω and array Λ (Ω, Λ=StableKeyValueSort(T,A)).

9. The method according to claim 8 , the encoding of the transformed data comprising:

viii) transforming Ω into a list Ω′ by storing only a number of occurrences of each appearing unique ion-mass (Ω′[i]={occs [i]; i€[0, M−1]}), where |Ω′|=M<<X n =|Ω|.

10. The method according to claim 9 , the encoding of the transformed data further comprising:

ix) computing an equivalent list Ω″ given as Ω″[i]=Ω″[i−1]+Ω′[i−1] by successively accumulating occurrences in Ω′, where Ω″ corresponds to start positions in Λ for each indexed unique ion-mass.

11. The method according to claim 9 , the transforming and encoding further comprising performing steps v)-viii) on all instances of N to construct the complete fragment-ion index.

12. The method according to claim 8 , the transforming of the data further comprising performing steps v)-vii) on all instances of N.

13. The method according to claim 8 , the encoding of the transformed data comprising using a running counter technique to compress repetitive and redundant information in the transformed data.

14. A system for compressing data to reduce index size, the system comprising:

a processor; and

a machine-readable medium in operable communication with the processor and comprising instructions stored thereon that, when executed by the processor, perform the following steps:

receiving an index comprising the data;

transforming the data to reduce its entropy and give transformed data; and

encoding the transformed data to reduce an amount of memory used by the transformed data and to give a custom data structure with reduced size compared to the received index,

the custom data structure being searchable without requiring any decompression prior to searching,

the index being a fragment-ion index and the data comprising mass spectrometry data,

the mass spectrometry data comprising mass spectrometry data of one or more peptides,

the transforming of the data comprising:

i) representing each theoretical mass spectrometry spectrum of the data as an ordered list of numbers where each number represents the m/z (mass divided by charge) of a predicted fragment-ion in the spectrum;

ii) ordering ions in each generated spectrum S i first by their ion-series (b/y), and then by their fragment charge, and then by their m/z values;

iii) grouping peptide sequences first by their length and then using all theoretical spectra in each group, where X represents the quantity of theoretical spectra in each group;

iv) constructing a matrix N=[S 1 , S 2 , . . . , S X ] as

N

=

[

i

11

¨

i

1

⁢

k

¨

i

1

⁢

n

i

j

⁢

⁢

1

¨

i

jk

¨

i

jn

i

X

⁢

⁢

1

¨

i

Xk

¨

i

Xn

]

,

where i jk corresponds to the m/z of k th fragment-ion in the j th spectrum, 0<i jk ≤(M−1), and M is maximum fragment-ion mass;

v) re-enumerating the ions to form a flattened list T=[i 0 , i 1 , i 2 , . . . , i Xn-1 ];

vi) defining a list A containing indices of ions in T, where A=[0, 1, 2, . . . , Xn −1 ], where a dimension of N is given as dim(N)=(X×n); and

vii) performing a Stable KeyValue Sort operation on A using T as a key to yield the transformed data in array Ω and array Λ (Ω, Λ=StableKeyValueSort(T,A)),

the encoding of the transformed data comprising:

viii) transforming Ω into a list Ω′ by storing only a number of occurrences of each appearing unique ion-mass (Ω′[i]={occs [i]; i€[0, M−1]}), where |Ω′|=M<<X n =|Ω|; and

ix) computing an equivalent list Ω″ given as Ω″[i]=Ω″[i−1]+Ω′[i−1] by successively accumulating occurrences in Ω′, where Ω″ corresponds to start positions in A for each indexed unique ion-mass, and

the transforming and encoding further comprising performing steps v)-ix) on all instances of N to construct the complete fragment-ion index.