Data consistency and rollback for cloud analytics
An extract-transform-load (ETL) platform fetches consistent datasets in a batch for a given period of time and provides the ability to rollback that batch. The batch may be fetched for an interval of time, and the ETL platform may fetch new or changed data from different cloud/on-premise applications. It will store this data in the cloud or on-premise to build data history. As the ETL platform fetches new data, the system will not overwrite existing data, but rather will create new versions so that change history is preserved. For any reason, if businesses would like to rollback data, they could rollback to any previous batch.
1. A method for retrieving consistent datasets, comprising:
collecting a first batch of data by a server from one or more tenant applications and associated with a first period of time, wherein the first batch of data includes one or more datasets;
updating the first batch of data in a batch log, wherein the updating occurs during and after the collection of the first batch of data;
storing the first batch of data in memory;
marking the first batch of data as the current batch of data in the batch log;
collecting a second batch of data by the server from the one or more tenant applications and associated with a second period of time subsequent to the first period of time, wherein the second batch of data includes one or more datasets, and wherein one or more the datasets of the second batch of data are distinct from the datasets of the first batch of data;
updating the second batch of data in the batch log, wherein the updating occurs during and after the collection of the second batch of data;
storing the second batch of data in memory;
marking the second batch of data as the current batch of data in the batch log;
detecting a rollback event indicating that a current dataset or a current batch of data should not be used;
marking the first batch of data as the current batch of data after the rollback event;
retrieving the first batch of data from memory using the batch log; and
overwriting the current batch of data corresponding to the second batch of data that should not be used with the retrieved first batch of data from memory.
2. The method of claim 1 , wherein the first batch of data and the second batch of data include the same data objects.
3. The method of claim 1 , wherein the batch log includes a unique identifier for each dataset associated with a batch of data.
4. The method of claim 1 , further comprising creating a new batch entry for each batch of data, wherein each new batch entry includes changed data between a current batch of data and a previous batch of data.
5. The method of claim 1 , further comprising logging a subsequent batch as failed when the subsequent batch fails to load into a collection server.
6. The method of claim 1 , wherein the rollback event is a request to perform a rollback to a previous batch.
7. The method of claim 1 , wherein the rollback event is automatically generated when data fails to load in a database.
8. A computer readable non-transitory storage medium having embodied thereon a program, the program being executable by a processor to perform a method for retrieving consistent datasets, the method comprising:
collecting a first batch of data by a server from one or more tenant applications and associated with a first period of time, wherein the first batch of data includes one or more datasets;
updating the first batch of data in a batch log, wherein the updating occurs during and after the collection of the first batch of data;
storing the first batch of data in memory;
marking the first batch of data as the current batch of data in the batch log;
collecting a second batch of data by the server from the one or more tenant applications and associated with a second period of time subsequent to the first period of time, wherein the second batch of data includes one or more datasets, and wherein one or more the datasets of the second batch of data are distinct from the datasets of the first batch of data;
updating the second batch of data in the batch log, wherein the updating occurs during and after the collection of the second batch of data;
storing the second batch of data in memory;
marking the second batch of data as the current batch of data in the batch log;
detecting a rollback event indicating that a current dataset or a current batch of data should not be used;
marking the first batch of data as the current batch of data after the rollback event;
retrieving the first batch of data from memory using the batch log; and
overwriting the current batch of data corresponding to the second batch of data that should not be used with the retrieved first batch of data from memory.
9. The computer readable non-transitory storage medium of claim 8 , wherein the first batch of data and the second batch of data include the same data objects.
10. The computer readable non-transitory storage medium of claim 8 , wherein the batch log includes a unique identifier for each dataset associated with a batch of data.
11. The computer readable non-transitory storage medium of claim 8 , further comprising creating a new batch entry for each batch of data, wherein each new batch entry includes changed data between a current batch of data and a previous batch of data.
12. The computer readable non-transitory storage medium of claim 8 , further comprising logging a subsequent batch as failed when the subsequent batch fails to load into a collection server.
13. The computer readable non-transitory storage medium of claim 8 , wherein the rollback event is a request to perform a rollback to a previous batch.
14. The computer readable non-transitory storage medium of claim 8 , wherein the rollback event is automatically generated when data fails to load in a database.
15. A system for retrieving consistent datasets, comprising:
a memory;
a processor; and
one or more modules stored in memory and executable by the processor to collect a first batch of data by a server from one or more tenant applications and associated with a first period of time, wherein the first batch of data includes one or more datasets, update the first batch of data in a batch log, wherein the updating occurs during and after the collection of the first batch of data, store the first batch of data in memory, mark the first batch of data as the current batch of data in the batch log, collect a second batch of data by the server from the one or more tenant applications and associated with a second period of time subsequent to the first period of time, wherein the second batch of data includes one or more datasets, and wherein one or more the datasets of the second batch of data are distinct from the datasets of the first batch of data, update the second batch of data in the batch log, wherein the updating occurs during and after the collection of the second batch of data, store the second batch of data in memory, mark the second batch of data as the current batch of data in the batch log, detect a rollback event indicating that a current dataset or a current batch of data should not be used, mark the first batch of data as the current batch of data after the rollback event, retrieve the first batch of data from memory using the batch log, and overwrite the current batch of data corresponding to the second batch of data that should not be used with the retrieved first batch of data from memory.