Efficient extraction of large data sets from a database
A method of efficiently extracting large data sets from a database includes receiving a query to be executed on a first view of one or more data sources, where previous result sets resulting from previous queries are stored separately from the one or more data sources. The method also includes receiving a current result set from the one or more data sources that is responsive to the query, where the query is reformatted before it is executed such that the result set does not overlap with the previous result sets. The method additionally includes generating a second view that combines the current result set and the one or more previous result sets, where the second view resolves changes in a schema of the one or data sources between when the previous result sets were received and when the current result set is received.
1. A method of efficiently extracting large data sets from data stores, the method comprising:
receiving a query to be executed on a first view of one or more data sources, wherein one or more queries on the first view have been previously executed prior to receiving the query, and one or more previous result sets resulting from the one or more queries are stored separately from the one or more data sources;
receiving a first schema for the one or more data sources prior to receiving the one or more previous result sets;
reformatting the query before the query is executed on the first view of the one or more data sources such that a current result set from the one or more data sources that is responsive to the reformatted query does not overlap with the one or more previous result sets;
receiving a second schema for the one or more data sources after receiving the one or more previous result sets
causing the reformatted query to be executed on the first view of the one or more data sources;
receiving the current result set from the one or more data sources responsive to the reformatted query; and
generating a second view that combines the current result set and the one or more previous result sets, wherein generating the second view comprises identifying a difference between the first schema and the second schema to generate code that creates one or more data tables that are referenced by the second view.
2. The method of claim 1 , wherein the current result set reorders columns that are in the one or more previous result sets.
3. The method of claim 1 , wherein the current result set adds an additional column that was not part of the one or more previous result sets.
4. The method of claim 3 , wherein the second view does not include the additional column.
5. The method of claim 1 , further comprising identifying the one or more previous result sets from among a plurality of previous result sets using an identifier for the first view.
6. The method of claim 5 , further comprising identifying the one or more previous result sets from among a plurality of previous result sets by grouping batches of result sets with same column headers.
7. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving a query to be executed on a first view of one or more data sources, wherein one or more queries on the first view have been previously executed prior to receiving the query, and one or more previous result sets resulting from the one or more queries are stored separately from the one or more data sources;
receiving a first schema for the one or more data sources prior to receiving the one or more previous result sets;
reformatting the query before the query is executed on the first view of the one or more data sources such that a current result set from the one or more data sources that is responsive to the reformatted query does not overlap with the one or more previous result sets;
receiving a second schema for the one or more data sources after receiving the one or more previous result sets
causing the reformatted query to be executed on the first view of the one or more data sources;
receiving the current result set from the one or more data sources responsive to the reformatted query; and
generating a second view that combines the current result set and the one or more previous result sets, wherein generating the second view comprises identifying a difference between the first schema and the second schema to generate code that creates one or more data tables that are referenced by the second view.
8. The non-transitory computer-readable medium of claim 7 , wherein the current result set reorders columns that are in the one or more previous result sets.
9. The non-transitory computer-readable medium of claim 7 , wherein the current result set adds an additional column that was not part of the one or more previous result sets.
10. The non-transitory computer-readable medium of claim 9 , wherein the second view does include the additional column.
11. The non-transitory computer-readable medium of claim 7 , wherein the operations further comprise:
identifying the one or more previous result sets from among a plurality of previous result sets using an identifier for the first view and by grouping batches of result sets with same column headers.
12. The non-transitory computer-readable medium of claim 7 , wherein the operations further comprise:
identifying the one or more previous result sets from among a plurality of previous result sets using an identifier for the first view.
13. A system comprising:
one or more processors, and
one or more memory devices comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving a query to be executed on a first view of one or more data sources, wherein one or more queries on the first view have been previously executed prior to receiving the query, and one or more previous result sets resulting from the one or more queries are stored separately from the one or more data sources;
receiving a first schema for the one or more data sources prior to receiving the one or more previous result sets;
reformatting the query before the query is executed on the first view of the one or more data sources such that a current result set from the one or more data sources that is responsive to the reformatted query does not overlap with the one or more previous result sets;
receiving a second schema for the one or more data sources after receiving the one or more previous result sets
causing the reformatted query to be executed on the first view of the one or more data sources;
receiving the current result set from the one or more data sources responsive to the reformatted query; and
generating a second view that combines the current result set and the one or more previous result sets, wherein generating the second view comprises identifying a difference between the first schema and the second schema to generate code that creates one or more data tables that are referenced by the second view.
14. The system of claim 13 , wherein the current result set reorders columns that are in the one or more previous result sets.
15. The system of claim 13 , wherein:
the current result set adds an additional column that was not part of the one or more previous result sets; and
the second view does not include the additional column.
16. The system of claim 13 , wherein the operations further comprise:
identifying the one or more previous result sets from among a plurality of previous result sets using an identifier for the first view and by grouping batches of result sets with same column headers.
17. The system of claim 13 , wherein the operations further comprise:
identifying the one or more previous result sets from among a plurality of previous result sets using an identifier for the first view.