Managing compression and storage of genomic data
A computer-implemented method according to one embodiment includes identifying genomic data within a system, dividing the genomic data into a plurality of partitions, creating a plurality of groups of different data types within each of the plurality of partitions, independently compressing, within each of the plurality of partitions, each of the plurality of groups of different data types to create a plurality of independently compressed partitions, validating each of the plurality of independently compressed partitions to create a plurality of validated independently compressed partitions, and saving the plurality of validated independently compressed partitions within the system.
1. A computer-implemented method, comprising:
identifying genomic data within a system;
dividing the genomic data into a plurality of partitions;
creating a plurality of groups of different data types within each of the plurality of partitions;
independently compressing, within each of the plurality of partitions, each of the plurality of groups of different data types to create a plurality of independently compressed partitions;
validating each of the plurality of independently compressed partitions to create a plurality of validated independently compressed partitions; and
saving the plurality of validated independently compressed partitions within the system.
2. The computer-implemented method of claim 1 , wherein identifying the genomic data includes determining a file type for the genomic data, a file extension for the genomic data, or a file type and a file extension for the genomic data.
3. The computer-implemented method of claim 1 , further comprising performing a preliminary compression on a portion of the genomic data to verify the genomic data.
4. The computer-implemented method of claim 1 , wherein the independent compressing is transparently performed within the system.
5. The computer-implemented method of claim 1 , wherein the plurality of groups of different data types are created by grouping similar types of data together within each of the plurality of partitions.
6. The computer-implemented method of claim 1 , wherein each of the plurality of partitions includes data of a plurality of different types.
7. The computer-implemented method of claim 1 , wherein the genomic data is grouped such that each of the plurality of groups of different data types contains data of a single type.
8. The computer-implemented method of claim 1 , wherein each of the plurality of groups of different data types is compressed using a compression algorithm different from the other groups, to create a plurality of compressed groups.
9. The computer-implemented method of claim 1 , wherein each compressed partition includes a plurality of compressed groups.
10. The computer-implemented method of claim 1 , wherein a compression algorithm used to compress each of the plurality of groups of different data types is specific to the genomic data stored within that group.
11. The computer-implemented method of claim 1 , wherein compression is performed within each of the plurality of partitions in parallel.
12. The computer-implemented method of claim 1 , wherein metadata is added to one or more of the plurality of independently compressed partitions, where the metadata for a compressed partition describes a location of the compressed partition within a sequence of the genomic data.
13. The computer-implemented method of claim 1 , wherein validating each of the plurality of independently compressed partitions is performed during the compressing of each of the plurality of independently compressed partitions.
14. The computer-implemented method of claim 1 , further comprising providing random access to one or more of the validated independently compressed partitions.
15. The computer-implemented method of claim 1 , wherein for each of the plurality of independently compressed partitions, validating each of the plurality of independently compressed partitions includes decompressing the independently compressed partition to create decompressed data, and comparing the decompressed data to an original partition before it was compressed.
16. The computer-implemented method of claim 15 , wherein the compressed partition is validated in response to determining that the decompressed data matches the original partition.
17. The computer-implemented method of claim 16 , further comprising hashing a validated compressed partition to create a hash value that is stored with the validated compressed partition.
18. The computer-implemented method of claim 15 , further comprising storing a corresponding original partition without being compressed in response to determining that the decompressed data does not match the original partition.
19. A computer program product for managing compression and storage of genomic data, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, the program instructions executable by a processor to cause the processor to perform a method comprising:
identifying, by the processor, genomic data within a system;
dividing, by the processor, the genomic data into a plurality of partitions;
creating, by the processor, a plurality of groups of different data types within each of the plurality of partitions;
independently compressing, by the processor within each of the plurality of partitions, each of the plurality of groups of different data types to create a plurality of independently compressed partitions;
validating, by the processor, each of the plurality of independently compressed partitions to create a plurality of validated independently compressed partitions; and
saving, by the processor, the plurality of validated independently compressed partitions within the system.
20. A system, comprising:
a processor; and
logic integrated with the processor, executable by the processor, or integrated with and executable by the processor, the logic being configured to:
identify genomic data within a system;
divide the genomic data into a plurality of partitions;
create a plurality of groups of different data types within each of the plurality of partitions;
independently compress, within each of the plurality of partitions, each of the plurality of groups of different data types to create a plurality of independently compressed partitions;
validate each of the plurality of independently compressed partitions to create a plurality of validated independently compressed partitions; and
save the plurality of validated independently compressed partitions within the system.