Providing data warehouse functionality on a local infrastructure
Providing data warehouse functionality on a local infrastructure, including: ingesting event data into a plurality of batches; applying, to ingested event data in the plurality of files, one or more transformations based on one or more queries applied to other event data stored in database tables of a data warehouse implemented in a data warehouse platform; and storing transformed event data as a plurality of files in storage independent of the data warehouse platform.
1 . A method of providing data warehouse functionality on a local infrastructure, the method comprising:
ingesting event data into a plurality of batches
storing the plurality of batches in memory;
applying, to ingested event data in the plurality of batches, one or more transformations based on one or more queries applied to second event data stored in database tables of a data warehouse implemented in a data warehouse platform, including:
accessing one or more logs of the data warehouse comprising the one or more queries; and
transforming the one or more queries to generate one or more second queries based on the one or more logs; and
storing transformed event data as a plurality of files in storage independent of the data warehouse platform.
2 . The method of claim 1 , wherein transforming the one or more queries comprises replacing values in the one or more queries with variables.
3 . The method of claim 1 , wherein the transformed event data is not stored in the storage as database records.
4 . The method of claim 1 , further comprising providing one or more of the ingested event data or the transformed event data to the data warehouse platform for storage in the data warehouse.
5 . The method of claim 4 , wherein providing one or more of the ingested event data or the transformed event data to the data warehouse platform includes providing one or more files to the data warehouse platform.
6 . The method of claim 1 , further comprising generating metadata describing the plurality of files.
7 . The method of claim 6 , further comprising storing the metadata in a metadata database.
8 . The method of claim 1 , further comprising performing one or more data analytics operations on at least a portion of the transformed event data by accessing one or more of the plurality of files.
9 . The method of claim 8 , wherein accessing the one or more of the plurality of files is based on metadata for the plurality of files.
10 . A computer readable medium including computer program instructions configurable to carry out the steps of:
ingesting event data into a plurality of batches;
storing the plurality of batches in memory;
applying, to ingested event data in the plurality of batches, one or more transformations based on one or more queries applied to second event data stored in database tables of a data warehouse implemented in a data warehouse platform, including:
accessing one or more logs of the data warehouse comprising the one or more queries; and
transforming the one or more queries to generate one or more second queries based on the one or more logs; and
storing transformed event data in a plurality of files in storage independent of the data warehouse platform.
11 . The computer readable medium of claim 10 wherein transforming the one or more queries comprises replacing values in the one or more queries with variables.
12 . The computer readable medium of claim 10 further including computer program instructions configurable to carry out the step of performing one or more data analytics operations on at least a portion of the transformed event data by accessing one or more of the plurality of files.
13 . The computer readable medium of claim 12 wherein accessing the one or more of the plurality of files is based on metadata for the plurality of files.
14 . The computer readable medium of claim 10 further including computer program instructions configurable to carry out the step of providing one or more of the ingested event data or the transformed event data to the data warehouse platform for storage in the data warehouse.
15 . The computer readable medium of claim 14 wherein providing one or more of the ingested event data or the transformed event data to the data warehouse platform includes providing one or more files to the data warehouse platform.
16 . The computer readable medium of claim 10 further including computer program instructions configurable to carry out the step of generating metadata describing the plurality of files.
17 . The computer readable medium of claim 16 further including computer program instructions configurable to carry out the step of storing the metadata in a metadata database.
18 . The computer readable medium of claim 10 wherein the transformed event data is not stored in the storage as database records.