Maintaining data consistency between transactional and non-transactional data stores

1. A method comprising:

receiving a request to archive a first set of data, the first set of data being stored in a transactional data store;

upon receipt of the request, archiving the first set of data, including copying the first set of data into an archive data store and inserting a data high water mark for the first set of data in the transactional data store and the archive data store, the data high water mark representing a time or order for archiving of the first set of data;

updating a reference high water mark for the transactional data store and the archive data store upon successful completion of operations for the archiving of the first set of data, the reference high water mark representing a time or order of completion of a last successful archive operation from the transactional data store to the archive data store;

receiving a query from a user for requested data, the query being either a request to the transactional data store or a request to the archive data store;

upon receiving the query for the requested data, comparing a current value of the reference high water mark and a data high water mark for the requested data in the transactional data store or the archive data store; and

determining whether to allow viewing of the requested data based at least in part on the comparison between the reference high water mark and the data high water mark for the requested data, including the following:

for a request to the transactional data store, excluding viewing of the requested data if the data high water mark for the requested data is less than or equal to the current value of the reference high water mark, and

for a request to the archive data store, excluding viewing of the requested data if the data high water mark for the requested data is greater than the current value of the reference high watermark.

2. The method of claim 1 , wherein determining whether to allow viewing of the requested data for a request to the transactional data store further includes allowing viewing of the requested data if a data high water mark has not been recorded for the requested data.

3. The method of claim 1 , further comprising rewriting the query for the requested data to implement data viewing protection.

4. The method of claim 3 , wherein rewriting the query includes inserting a reference to the current value of the reference high water mark in the query.

5. The method of claim 1 , further comprising deleting the first set of data asynchronously from the transactional data store subsequent to the successful completion of the archive operation.

6. A system comprising:

a transactional data store including a first set of storage elements to store data;

an archive data store including a second set of storage elements to store data;

an archive engine to archive a first set of data from the transactional data store to the archive data store in response to an archival request, the archive engine to:

copy the first set of data into the archive data store and insert a data high water mark for the first set of data in the transactional data store and the archive data store, the data high water mark representing a time or order for archiving of the first set of data, and

update a reference high water mark upon successful completion of the archiving of the first set of data, the reference high water mark representing a time or order of completion of a last successful archive operation from the transactional data store to the archive data store; and

a transaction engine to implement data transactions in the transactional data store and an archive retrieval engine to retrieve data from the archive data store;

wherein, upon receiving a query from a user for requested data, the query being either a request to the transactional data store or a request to the archive data store, the transaction engine or the archive retrieval engine is to:

compare a current value of the reference high water mark and a data high water mark for the requested set of data in the transactional data store or the archive data store;

determine whether to allow viewing of the requested data based at least in part on the comparison between the reference high water mark and the data high water mark for the requested set of data, including the following:

for a request to the transactional data store, exclude viewing of the requested data if the data high water mark for the requested data is less than or equal to the current value of the reference high water mark, and

for a request to the archive data store, exclude viewing of the requested data if the data high water mark for the requested data is greater than the current value of the reference high watermark.

7. The system of claim 6 , wherein determining whether to allow viewing of the requested data for a request to the transactional data store further includes the transaction engine to allow viewing of the requested data if a data high water mark has not been recorded for the requested data.

8. The system of claim 6 , wherein the transaction engine or the archive retrieval engine are to rewrite the query for the requested data to implement data viewing protection.

9. The system of claim 8 , wherein rewriting the query includes inserting a reference to the current value of the reference high water mark in the query.

10. The system of claim 6 , wherein the transaction engine is to delete the first set of data asynchronously from the transactional data store subsequent to the successful completion of the archive operation.

11. The system of claim 6 , wherein the system provides a consistent view of the first set of data that is tolerant of intermittent failures during archive and delete operations.

12. The system of claim 6 , wherein the data high water mark includes a date and time stamp.

13. The system of claim 6 , wherein the data high water mark includes an accession sequence number.

14. A non-transitory computer-readable storage medium having stored thereon data representing sequences of instructions that, when executed by a processor, cause the processor to perform operations comprising:

receiving a request to archive a first set of data, the first set of data being stored in a first data store;

upon receipt of the request, archiving the first set of data, including copying the first set of data into a second data store and inserting a data high water mark for the first set of data in the first data store and the second data store, the data high water mark representing a time or order for archiving of the first set of data;

updating a reference high water mark for the first data store and the second data store upon successful completion of operations for the archiving of the first set of data, the reference high water mark representing a time or order of completion of a last successful archive operation from the first data store to the second data store;

receiving a query from a user for requested data, the query being either a request to the first data store or a request to the second data store;

upon receiving the query for the requested data, comparing a current value of the reference high water mark and a data high water mark for the requested data in the transactional data store or the archive data store; and

determining whether to allow viewing of the requested data based at least in part on the comparison between the reference high water mark and the data high water mark for the requested data, including the following:

for a request to the first data store, excluding viewing of the requested data if the data high water mark for the requested data is less than or equal to the current value of the reference high water mark, and

for a request to the second data store, excluding viewing of the requested data if the data high water mark for the requested data is greater than the current value of the reference high watermark.

15. The storage medium of claim 14 , wherein determining whether to allow viewing of the requested data for a request to the first data store further includes allowing viewing of the requested data if a data high water mark has not been recorded for the requested data.

16. The storage medium of claim 14 , further comprising instructions that, when executed by the processor, cause the processor to perform operations comprising:

rewriting the query for the requested data to implement data viewing protection.

17. The storage medium of claim 16 , wherein rewriting the query includes inserting a reference to the current value of the reference high water mark in the query.

18. The storage medium of claim 14 , further comprising instructions that, when executed by the processor, cause the processor to perform operations comprising:

deleting the first set of data asynchronously from the first data store subsequent to the successful completion of the archive operation.