In-database single-nucleotide genetic variant analysis
Genetic data in row-wise flat files, such as VCF and VCF-like files, comprising a plurality of data elements of different types is analyzed using a parallel framework in an MPP shared-nothing distributed database having a plurality of distributed segments by first parsing the data into groups of data elements of the same types, converting the data into entry-wise genetic data such that the same types of data elements are in a column, and distributing and storing the entry-wise genetic data in the distributed segments. SQL database queries are used to analyze the genetic data, including locating probable significant associations between genotype and phenotype data.
1. A method of analyzing genetic data using a distributed parallel framework, comprising:
receiving row-wise data, each row of the row-wise data storing an identifier of a respective individual and characteristics of the respective individual under respective column names;
converting the row-wise data into entry-wise tables each organized in a plurality of columns and a plurality of rows, wherein the converting comprises:
parsing, based on the column names, each row of the row-wise data into genetic data representing genetic variants of the respective individual and non-genetic data representing covariate data of the respective individual;
storing the genetic data of each row of the row-wise data as a plurality of rows in a first entry-wise table, wherein a portion of the column names in the row-wise data corresponding to the genetic variants are inserted as values in a plurality of rows in the first entry-wise table, wherein the parsing comprises:
grouping the first entry-wise table by identifiers of the genetic variant or individuals; and
grouping the second entry-wise table by genotype or individuals, and
wherein the first entry-wise table is indexed at least by identifiers of the genetic variants, and the second entry-wise table is indexed at least by values of the covariate data; and
storing the non-genetic data of each row of the row-wise data as a plurality of rows in a second entry-wise table, wherein a portion of the column names in the row-wise data corresponding to the covariate data are inserted as values in a plurality of rows in the second entry-wise table;
distributing the first and second entry-wise tables across a plurality of database segments of a parallel distributed database; and
running database queries in parallel on the distributed first and second entry-wise tables by joining the first and second entry-wise tables by the identifier of the individual.
2. The method of claim 1 , wherein the row-wise data is stored in one or more variant cell format (VCF) row-wise files, and the column names of the row-wise data are fields in the VCF row-wise files, the fields representing one or more data element types.
3. The method of claim 1 , wherein the genetic variants includes single-nucleotide polymorphism (SNP).
4. The method of claim 3 , comprising storing the first and second entry-wise tables separately on the database segments.
5. The method of claim 4 further comprising analyzing said genetic data on different combinations of the non-genetic data using SQL queries.
6. The method of claim 1 , further comprising computing, in parallel across the plurality of database segments, probabilities of genetic variants being associated with a particular disease by analyzing the genetic data and non-genetic data in the first and second entry-wise tables against data related to said disease in a disease table.
7. The method of claim 1 further comprising analyzing the genetic data by running statistical analytics in parallel on the genetic variants in said stored first entry-wise table.
8. The method of claim 1 further comprising randomly swapping identifiers of rows of the first and second entry-wise tables, and running permutation tests on the first and second entry-wise data tables based upon said identifiers.
9. A non-transitory computer readable medium for storing executable instructions for controlling operation of a computer to analyze genetic data using a distributed parallel framework, comprising instructions for:
receiving row-wise data, each row of the row-wise data storing an identifier of a respective individual and characteristics of the respective individual under respective column names;
converting the row-wise data into entry-wise tables each organized in a plurality of columns and a plurality of rows, wherein the converting comprises:
parsing, based on the column names, each row of the row-wise data into genetic data representing genetic variants of the respective individual and non-genetic data representing covariate data of the respective individual;
storing the genetic data each row of the row-wise data as a plurality of rows in a first entry-wise table, wherein a portion of the column names in the row-wise data corresponding to the genetic variants are inserted as values in a plurality of rows in the first entry-wise table; and
storing the non-genetic data of each row of the row-wise data as a plurality of rows in a second entry-wise table, wherein a portion of the column names in the row-wise data corresponding to the covariate data are inserted as values in a plurality of rows in the second entry-wise table, wherein the parsing comprises:
grouping the first entry-wise table by identifiers of the genetic variant or individuals; and
grouping the second entry-wise table by genotype or individuals, and
wherein the first entry-wise table is indexed at least by identifiers of the genetic variants, and the second entry-wise table is indexed at least by values of the covariate data;
distributing the first and second entry-wise tables across a plurality of database segments of a parallel distributed database; and
running database queries in parallel on the distributed first and second entry-wise tables by joining the first and second entry-wise tables by the identifier of the individual.
10. The computer readable medium of claim 9 , wherein the row-wise data is stored in one or more variant cell format (VCF) row-wise files, and the column names of the row-wise data are fields in the VCF row-wise files, the fields representing one or more data element types.
11. The computer readable medium of claim 10 , wherein the genetic variant includes single-nucleotide polymorphism (SNP).
12. The computer readable medium of claim 11 , wherein the operations comprise storing the first and second entry-wise tables separately on the database segments.
13. The computer readable medium of claim 12 , the operations further comprising, in parallel across the plurality of database segments, probabilities of genetic variants being associated with a particular disease by analyzing the genetic data and non-genetic data in the first and second entry-wise tables against data related to said disease in a disease table.
14. The computer readable medium of claim 12 , the operations further comprising instructions for analyzing said genetic data on different combinations of the non-genetic data using SQL queries.
15. The computer readable medium of claim 9 , the operations further comprising analyzing the genetic data by running statistical analytics in parallel on the genetic variants in said stored first entry-wise table.
16. The computer readable medium of claim 9 , the operations further comprising randomly swapping identifiers of rows of the first and second entry-wise tables, and for running permutation tests on the first and second entry-wise data tables based upon said identifiers.
17. A system comprising:
one or more processors;
a non-transitory computer-readable medium for storing executable instructions operable to cause the one or more processors to perform operations comprising:
receiving row-wise genetic data, each row of the row-wise data storing an identifier of a respective individual and characteristics of the respective individual under respective column names;
converting the row-wise data into entry-wise tables each organized in a plurality of columns and a plurality of rows, wherein the converting comprises:
parsing, based on the column names, each row of the row-wise data into genetic data representing genetic variants of the respective individual and non-genetic data representing covariate data of the respective individual;
storing the genetic data of each row of the row-wise data as a plurality of rows in a first entry-wise table, wherein a portion of the column names in the row-wise data corresponding to the genetic variants are inserted as values in a plurality of rows in the first entry-wise table, wherein the parsing comprises:
grouping the first entry-wise table by identifiers of the genetic variant or individuals; and
grouping the second entry-wise table by genotype or individuals, and
wherein the first entry-wise table is indexed at least by identifiers of the genetic variants, and the second entry-wise table is indexed at least by values of the covariate data; and
storing the non-genetic data of each row of the row-wise data as a plurality of rows in a second entry-wise table, wherein a portion of the column names in the row-wise data corresponding to the covariate data are inserted as values in a plurality of rows in the second entry-wise table;
distributing the first and second entry-wise tables across a plurality of database segments of a parallel distributed database; and
running database queries in parallel on the distributed first and second entry-wise tables by joining the first and second entry-wise tables by the identifier of the individual.
18. The system of claim 17 , wherein the row-wise genetic data is stored in one or more variant cell format (VCF) row-wise files, and the column names of the row-wise genetic data are fields in the VCF row-wise files, the fields representing one or more data element types.