FastQ/FastA compression systems and methods
Systems and methods to analyze and significantly compress FastQ and/or FastA datasets are disclosed. The methodology includes algorithms to compress sequences, quality scores and identifiers of read files. The method relies on reducing the dimension and redundancy in genomic data in a unique and optimal way and in the binary format. The methodology also includes the decoding protocols to decompress the compressed data with zero loss.
1. A method for data compression of genomic data comprising:
receiving a data file comprised of sequence bases, quality scores, and identifiers, the sequence bases comprising regular bases and irregular bases, the regular bases comprising adenine (A), cytosine (C), guanine (G), and thymine (T);
applying an optimization algorithm to the quality scores, sequence k-mers from the sequence bases, and the identifiers;
splitting long reads of the sequence bases and the quality scores into smaller segments;
deleting duplicated and semi-duplicated reads for the sequence bases and the quality scores;
performing a dimensionality reduction on the sequence bases;
storing a template of the identifier that is consistent across the data file;
detecting and storing the location and type of each irregular base;
encoding the data file in a binary format such that each regular base is represented by one of the four two-digit binary numbers and all irregular bases are represented by one of the four two-digit binary numbers; and
compressing the encoded data file.
2. The method of claim 1 , wherein the identifiers are comprised of sequencing run data and cluster data.
3. The method of claim 1 , wherein the quality scores comprise the sequence of quality values for the sequence bases.
4. The method of claim 1 , wherein:
the irregular bases include N and a plurality of additional irregular bases; and
storing the location and type of each irregular base comprises storing the location of each irregular base and the type of each additional irregular base.
5. The method of claim 4 , wherein dimensionality reduction is performed using binary labels from a k-mers dictionary and the base sequences.
6. The method of claim 1 , wherein the data file is a FastQ file or a FastA file.
7. The method of claim 1 , wherein the optimization algorithm determines an optimal value for encoding algorithm hyper-parameters, the sequence k-mers from the bases, and the identifiers.
8. The method of claim 1 , wherein the optimization algorithm is a function of protocol hyperparameters and distribution indices of the data file.
9. The method of claim 1 , wherein the compression is reference-free.
10. The method of claim 1 , wherein the compression is tuned based on characteristics of the data file.
11. A non-transitory computer storage medium that stores a program thereon that causes a computer to execute a process comprising:
receiving a data file comprised of sequence bases, quality scores, and identifiers, the sequence bases comprising regular bases and irregular bases, the regular bases comprising adenine (A), cytosine (C), guanine (G), and thymine (T);
applying an optimization algorithm to the quality scores, sequence k-mers from the sequence bases, and the identifiers;
splitting long reads of the sequence bases and the quality scores into smaller segments;
deleting duplicated and semi-duplicated reads for the sequence bases and the quality scores;
performing a dimensionality reduction on the sequence bases;
storing a template of the identifier that is consistent across the data file;
detecting and storing the location and type of each irregular base;
encoding the data file in a binary format such that each regular base is represented by one of the four two-digit binary numbers and all irregular bases are represented by one of the four two-digit binary numbers; and
compressing the data file, wherein the compression is lossless.
12. The non-transitory computer medium of claim 11 , wherein the identifiers are comprised of sequencing run data and cluster data.
13. The non-transitory computer medium of claim 11 , wherein the quality scores comprise the sequence of quality values for the sequence bases.
14. The non-transitory computer medium of claim 11 , wherein:
the irregular bases include N and a plurality of non-N irregular bases; and
storing the location and type of each irregular base comprises storing the location of each irregular base and the type of each non-N irregular base.
15. The non-transitory computer medium of claim 14 , wherein dimensionality reduction is performed using binary labels from a k-mers dictionary and the regular bases.
16. The non-transitory computer medium of claim 11 , wherein the data file is a FastQ file or a FastA file.
17. The non-transitory computer medium of claim 11 , wherein the optimization algorithm determines the optimal hyperparameters values for each of the quality scores, the sequence k-mers from the bases, and the identifiers.
18. The non-transitory computer medium of claim 11 , wherein the optimization algorithm is a function of protocol hyperparameters and distribution indices of the data file.
19. The non-transitory computer medium of claim 11 , wherein the compression is reference-free.
20. The non-transitory computer medium of claim 11 , wherein the compression is tuned based on characteristics of the data file.