System and method for data storage
One embodiment operates a computer system for storing and retrieving data from a data store. The computer system has a data manager device operable to manage patterns in the stored data along with pointers relating to the patterns in a manner which enables reconstruction of the stored data in original form.
1. A method comprising:
transferring, by a computing device, an extract of data from a data repository to an archive at a point T in time, wherein the data is represented by a first schema at time T;
storing, by the computing device, the extract of data as a combination of (i) a header portion (ii) an index portion and (iii) a pointer to a pattern space in the archive, the pattern space comprising at least one pattern,
wherein the header portion contains information on how to interpret the at least one pattern in the pattern space in order to recover the data in the form it existed in the data repository, and the index portion identifies at least one pattern from within the pattern space that represent entire data records from the data repository;
receiving, by the computing device, a data query at a time later than T wherein the data schema in the data repository has changed to a second schema which differs from the first schema; and
returning, by the computing device, in response to said data query, data from the archive utilising data in said header portion, index portion and pattern space such that the first schema behaves in response to the query as if it had been created under the second schema.
2. The method of claim 1 , a transformation on data is performed during processing the data query.
3. The method of claim 2 , wherein the transformation is selected from one or more of: dropping a column, renaming a column, adding a column, re-typing a column, querying at a point in time, splitting a column, and merging a column.
4. The method of claim 2 , wherein the extract of data is stored as immutable data.
5. The method of claim 1 , wherein the header portion comprises at least (i) names of fields that appear in records of a table in the data repository and (ii) types of these fields.
6. The method of claim 1 , wherein the data repository is a relational database.
7. The method of claim 1 , wherein the data repository is a data warehouse.
8. The method of claim 1 , wherein the data repository is a star schema database.