FORMATTING AND STORAGE OF GENETIC MARKERS
The disclosed embodiments concern methods, apparatus, systems, and computer program products for storing and retrieving genetic data for individuals. In some implementations, a storage format is provided that allows genetic data to be defined by metadata for reproduce-ability.
1 . A method of providing information from one or more files of a database having a table with at least about 20M columns, the method comprising:
receiving a request for a plurality of markers for each individual of about 10,000 or more individuals, wherein the plurality of markers is at least about 10,000 markers;
retrieving metadata corresponding to a plurality of files, wherein each file stores a continuous set of markers for a batch of samples, and wherein the continuous set of markers is less than about 10,000 markers;
accessing the plurality of files based on the request and the metadata corresponding to the plurality of files; and
providing the plurality of markers for each of the about 10,000 or more individuals.
2 . The method of claim 1 , wherein the request further comprises a datetime, and wherein the method further comprises:
identifying a first set of samples that are prior to the datetime; and
providing the plurality of markers for each of the one or more individuals based on the first set of samples.
3 . The method of claim 2 , wherein the first set of samples comprises a first sample and a second sample, the first sample and the second sample corresponding to a first individual of the one or more individuals, and wherein the method further comprises:
determining that the second sample was determined later than the first sample; and
providing the plurality of markers for the first individual based on the second sample.
4 . The method of claim 1 , wherein the batch of samples is at least 100,000 samples.
5 . The method of claim 1 , wherein the files are parquet files.
6 . The method of claim 1 , wherein the plurality of markers is at least about 1 million markers.
7 . A system to provide information from one or more files of a database having a table with at least about 20M columns, the system comprising one or more processors and one or more memory devices configured to:
receive a request for a plurality of markers for each individual of about 10,000 or more individuals, wherein the plurality of markers is at least about 10,000 markers;
retrieve metadata corresponding to a plurality of files, wherein each file stores a continuous set of markers for a batch of samples, and wherein the continuous set of markers is less than about 10,000 markers;
access the plurality of files based on the request and the metadata corresponding to the plurality of files; and
provide the plurality of markers for each of the about 10,000 or more individuals.
8 . The system of claim 7 , wherein the request further comprises a datetime, and wherein the one or more processors and one or more memory devices are further configured to:
identify a first set of samples that are prior to the datetime; and
provide the plurality of markers for each of the one or more individuals based on the first set of samples.
9 . The system of claim 8 , wherein the first set of samples comprises a first sample and a second sample, the first sample and the second sample corresponding to a first individual of the one or more individuals, and wherein the one or more processors and one or more memory devices are further configured to:
determine that the second sample was determined later than the first sample; and
provide the plurality of markers for the first individual based on the second sample.
10 . The system of claim 7 , wherein the batch of samples is at least 100,000 samples.
11 . The system of claim 7 , wherein the files are parquet files.
12 . The system of claim 7 , wherein the plurality of markers is at least about 1 million markers.
13 . A non-transient computer-readable medium comprising program instructions that, when executed by one or more processors, cause the one or more processors to:
receive a request for a plurality of markers for each individual of about 10,000 or more individuals, wherein the plurality of markers is at least about 10,000 markers;
retrieve metadata corresponding to a plurality of files, wherein each file stores a continuous set of markers for a batch of samples, and wherein the continuous set of markers is less than about 10,000 markers;
access the plurality of files based on the request and the metadata corresponding to the plurality of files; and
provide the plurality of markers for each of the about 10,000 or more individuals.
14 . The non-transient computer-readable medium of claim 13 , wherein the request further comprises a datetime, and wherein the program instructions comprise further instructions that, when executed by one or more processors, cause the one or more processors to:
identify a first set of samples that are prior to the datetime; and
provide the plurality of markers for each of the one or more individuals based on the first set of samples.
15 . The non-transient computer-readable medium of claim 14 , wherein the first set of samples comprises a first sample and a second sample, the first sample and the second sample corresponding to a first individual of the one or more individuals, and wherein the program instructions comprise further instructions that, when executed by one or more processors, cause the one or more processors to:
determine that the second sample was determined later than the first sample; and
provide the plurality of markers for the first individual based on the second sample.
16 . The non-transient computer-readable medium of claim 13 , wherein the batch of samples is at least 100,000 samples.
17 . The non-transient computer-readable medium of claim 13 , wherein the files are parquet files.
18 . The non-transient computer-readable medium of claim 13 , wherein the plurality of markers is at least about 1 million markers.
19 . The non-transient computer-readable medium of claim 13 , wherein the plurality of markers is associated with a plurality of genomic regions, and wherein each genomic region is responsible for storing one or more marker-major statistics.
20 . The non-transient computer-readable medium of claim 19 , wherein the marker-major statistics comprise contiguous marker locations for a chromosome or group of chromosomes.