Common extract store
Approaches for a persistent data store that may be used to service data queries originating from data warehousing applications, operational reporting applications, or data archiving and data feed applications. At periodic intervals, data may be extracted from one or more disparate data sources. The extracted data may be stored in the persistent data store in a tabular format. A backward delta data set is generated. The backward delta data set identifies what additions, deletions, and modifications need to be made to the data stored in the persistent data store to return the data to a most recent version. Backward delta data sets are stored and maintained in accordance with a policy that instructs two or more backward delta data sets, associated with a continuous interval of time, be combined into a single backward delta data set after the expiration of a specified amount of time.
1. A non-transitory computer-readable storage medium storing one or more sequences of instructions for providing a persistent data store, which when executed by one or more processors, causes:
at predetermined points in time, extracting a data set from one or more data sources, wherein the extracting comprises converting the extracted data set into one or more tables, and concatenating and storing the one or more tables of the extracted data set in the persistent data store in a single logical table having a tabular format, wherein the tabular format comprises a logical representation of a current version of the data set available within one or more data sources, and wherein storing the extracted data set comprises:
generating a backward delta data set that identifies a set of operations, including any additions, deletions, and modifications, which need to be made to data stored in the tabular format of the data store to cause a current version of the persistent data store to return to a most recent version of the persistent data store, and
storing and maintaining the backward delta data set in accordance with a policy that instructs two or more backward delta data sets, associated with a continuous interval of time, be combined into a single backward delta data set after the expiration of a specified amount of time;
upon receiving a query to be performed against a particular state of the data store at a particular point in time, (a) identifying a set of backward delta data sets associated with all time periods at least as recent as the particular point in time, (b) applying the set of backward delta data sets to the current version of the data store to obtain a historical version of the data store, and (c) performing the query against the historical version of the data store; and
upon receiving a query for data satisfying a set of criteria to be performed against a particular version of a schema, without human intervention:
consulting a metadata that identifies a mapping between the tabular format of the persistent data store and the particular version of a schema, and
converting, using the metadata and without human intervention, the tabular format of the persistent data store into a converted data set in the particular version of the schema and performing the query against the converted data set.
2. The non-transitory computer-readable storage medium of claim 1 , wherein the data store is used to service data queries originating from (a) one or more data warehousing applications, (b) one or more operational reporting applications, and (c) one or more data archiving, and (d) one or more data feed applications.
3. The non-transitory computer-readable storage medium of claim 1 , wherein execution of the one or more sequences of instructions further causes:
upon determining that the extracted data set indicates a column in a particular data source has been removed, (a) updating the tabular format of the data store to remove a corresponding column associated with the removed column, and (b) configuring the backward delta data set associated with the extracted data set to add the corresponding column back to the tabular format of the data store; and
upon determining that the extracted data set indicates a new column in the particular data source has been added, (a) updating the tabular format of the data store to add a new corresponding column associated with the new column, and (b) configuring the backward delta data set associated with the extracted data set to remove the new corresponding column from the tabular format of the data store.
4. The non-transitory computer-readable storage medium of claim 1 , wherein all historical data reflecting any prior versions of data stored by the persistent data store is recorded in one or more backward delta data sets.
5. The non-transitory computer-readable storage medium of claim 1 , wherein the storing and maintaining the backward delta data set is performed without human intervention by one or more software processes.
6. An apparatus providing a persistent data store, comprising:
one or more processors; and
one or more sequences of instructions, which when executed by the one or more processors, causes:
at predetermined points in time, extracting a data set from one or more data sources, wherein the extracting comprises converting the extracted data set into one or more tables, and concatenating and storing the one or more tables of the extracted data set in the persistent data store in a single logical table having a tabular format, wherein the tabular format comprises a logical representation of a current version of the data set available within one or more data sources, and wherein storing the extracted data set comprises:
generating a backward delta data set that identifies a set of operations, including any additions, deletions, and modifications, which need to be made to data stored in the tabular format of the data store to cause a current version of the persistent data store to return to a most recent version of the persistent data store, and
storing and maintaining the backward delta data set in accordance with a policy that instructs two or more backward delta data sets, associated with a continuous interval of time, be combined into a single backward delta data set after the expiration of a specified amount of time;
upon receiving a query to be performed against a particular state of the data store at a particular point in time, (a) identifying a set of backward delta data sets associated with all time periods at least as recent as the particular point in time, (b) applying the set of backward delta data sets to the current version of the data store to obtain a historical version of the data store, and (c) performing the query against the historical version of the data store; and
upon receiving a query for data satisfying a set of criteria to be performed against a particular version of a schema, without human intervention:
consulting a metadata that identifies a mapping between the tabular format of the persistent data store and the particular version of a schema, and
converting, using the metadata and without human intervention, the tabular format of the persistent data store into a converted data set in the particular version of the schema and performing the query against the converted data set.
7. The apparatus of claim 6 , wherein the data store is used to service data queries originating from (a) one or more data warehousing applications, (b) one or more operational reporting applications, and (c) one or more data archiving, and (d) one or more data feed applications.
8. The apparatus of claim 6 , wherein execution of the one or more sequences of instructions further causes:
upon determining that the extracted data set indicates a column in a particular data source has been removed, (a) updating the tabular format of the data store to remove a corresponding column associated with the removed column, and (b) configuring the backward delta data set associated with the extracted data set to add the corresponding column back to the tabular format of the data store; and
upon determining that the extracted data set indicates a new column in the particular data source has been added, (a) updating the tabular format of the data store to add a new corresponding column associated with the new column, and (b) configuring the backward delta data set associated with the extracted data set to remove the new corresponding column from to the tabular format of the data store.
9. The apparatus of claim 6 , wherein all historical data reflecting any prior versions of data stored by the persistent data store is recorded in one or more backward delta data sets.
10. The apparatus of claim 6 , wherein the storing and maintaining the backward delta data set is performed without human intervention by one or more software processes.
11. A method for providing a persistent data store, comprising:
at predetermined points in time, extracting a data set from one or more data sources, wherein the extracting comprises converting the extracted data set into one or more tables, and concatenating and storing the one or more tables of the extracted data set in the persistent data store in a single logical table having a tabular format, wherein the tabular format comprises a logical representation of a current version of the data set available within one or more data sources, and wherein storing the extracted data set comprises:
generating a backward delta data set that identifies a set of operations, including any additions, deletions, and modifications, which need to be made to data stored in the tabular format of the data store to cause a current version of the persistent data store to return to a most recent version of the persistent data store, and
storing and maintaining the backward delta data set in accordance with a policy that instructs two or more backward delta data sets, associated with a continuous interval of time, be combined into a single backward delta data set after the expiration of a specified amount of time;
upon receiving a query to be performed against a particular state of the data store at a particular point in time, (a) identifying a set of backward delta data sets associated with all time periods at least as recent as the particular point in time, (b) applying the set of backward delta data sets to the current version of the data store to obtain a historical version of the data store, and (c) performing the query against the historical version of the data store; and
upon receiving a query for data satisfying a set of criteria to be performed against a particular version of a schema, without human intervention:
consulting a metadata that identifies a mapping between the tabular format of the persistent data store and the particular version of a schema, and
converting, using the metadata and without human intervention, the tabular format of the persistent data store into a converted data set in the particular version of the schema and performing the query against the converted data set.
12. The method of claim 11 , wherein the data store is used to service data queries originating from (a) one or more data warehousing applications, (b) one or more operational reporting applications, and (c) one or more data archiving, and (d) one or more data feed applications.
13. The method of claim 11 , further comprising:
upon determining that the extracted data set indicates a column in a particular data source has been removed, (a) updating the tabular format of the data store to remove a corresponding column associated with the removed column, and (b) configuring the backward delta data set associated with the extracted data set to add the corresponding column back to the tabular format of the data store; and
upon determining that the extracted data set indicates a new column in the particular data source has been added, (a) updating the tabular format of the data store to add a new corresponding column associated with the new column, and (b) configuring the backward delta data set associated with the extracted data set to remove the new corresponding column from to the tabular format of the data store.
14. The method of claim 11 , wherein all historical data reflecting any prior versions of data stored by the persistent data store is recorded in one or more backward delta data sets.
15. The method of claim 11 , wherein the storing and maintaining the backward delta data set is performed without human intervention by one or more software processes.