Method and system for no downtime, initial data upload for real-time, continuous data protection
View Patent ↗A data management system or “DMS” provides an automated, continuous, real-time, substantially no downtime data protection service to one or more data sources associated with a set of application host servers. To facilitate the data protection service, a host driver embedded in an application server captures real-time data transactions, preferably in the form of an event journal that is provided to other DMS components. The driver functions to translate traditional file/database/block I/O and the like into a continuous, application-aware, output data stream. The host driver includes an event processor. When a data protection command for a given data source is forwarded to a host driver, the event processor enters into an initial upload state. During this state, the event processor gathers a list of data items of the data source to be protected and creates a data list. Then, the event processor moves the data (as an upload, preferably one data element at a time) to a DMS core to create initial baseline data. In an illustrative embodiment, the upload is a stream of granular application-aware data chunks that are attached to upload events. Simultaneously, while the baseline is uploading and as the application updates the data on the host, checkpoint granular data, metadata, and data events are continuously streamed into the DMS core. During this upload phase, the application does not have to be shutdown.
1. A method of providing a data service, comprising:
ordering into a given data structure a set of data items associated with a given application executing in a first processing environment;
setting a pointer at an initial position with respect to the given data structure and flagging the set of data items for a data transfer;
monitoring events as the given application interfaces to a local data store and collecting a set of events; and
for a given event, wherein the given event is a given type of event, processing the given event against the given data structure by the following sub-steps:
locating a data item at a then-current position of the pointer;
determining whether the data item is flagged for data transfer;
if the data item is flagged for upload, retrieving given information from the given data structure;
forwarding the given information for inclusion in an application-aware data stream;
unflagging the data item for data transfer; and
resetting the pointer to a next data item in the given data structure.
2. A method of providing a data service, comprising:
ordering into a given data structure a set of data items associated with a given application executing in a first processing environment;
setting a pointer at an initial position with respect to the given data structure and flagging the set of data items for a data transfer;
monitoring events as the given application interfaces to a local data store and collecting a set of events; and
for a given event, wherein the given event is a given type of event, processing the given event against the given data structure by the following sub-steps:
locating a given data item in the given data structure;
determining whether the given data item is positioned above a then-current position of the pointer;
if the given data item is not positioned above the then-current position of the pointer, placing given information in the given data structure; and
dropping the event.
3. The method as described in claim 2 wherein if the given data item is positioned above the then-current position of the pointer, further including the step of:
transferring data associated with the given data item to a remote data store without additional processing.
4. A system for protecting a data source associated with a host in a first processing environment, comprising:
a processor;
a sorted source tree having a set of data items associated with the data source, the sorted source tree having a pointer associated therewith;
code executable in the processor for flagging the set of data items for a data transfer;
code executable in the processor for monitoring and capturing real-time events as the data source interfaces to a local data store in the first processing environment; and
code executable in the processor for processing the set of data items in the sorted source tree as follows:
(a) determining if a real-time event is then being captured;
(b) if a real-time event is not then being captured, locating a data item in the sorted source tree based on a position of the pointer, placing the data item into an application-aware data stream, unflagging the data item, and positioning the pointer at a next position in the sorted source tree; and
(c) if a real-time event is then being captured, locating an associated data item in the sorted source tree based on the position of the pointer, and determining if the associated data item is unflagged, wherein:
(i) if the associated data item is unflagged, processing the real-time event and inserting information associated with such processing into the application-aware data stream; and
(ii) if the associated data item is not flagged, dropping the real-time event.
5. The system as described in claim 4 further including code executable in the processor for transferring the application-aware data stream from the first processing environment to a second processing environment.
6. The system as described in claim 4 wherein a given data item in the set of data items is associated with data that is being modified by the data source.
7. A method of providing a data service, comprising:
ordering into a data structure a set of data items associated with an application executing in a first processing environment, the data structure having a pointer associated therewith;
flagging the set of data items for a data transfer;
monitoring and capturing real-time events as the application interfaces to a local data store in the first processing environment; and
processing the set of data items in the data structure as follows:
(a) determining if a real-time event is then being captured;
(b) if a real-time event is not then being captured, locating a data item in the data structure based on a position of the pointer, placing the data item into an application-aware data stream, unflagging the data item, and positioning the pointer at a next position in the data structure; and
(c) if a real-time event is then being captured, locating an associated data item in the data structure based on the position of the pointer, and determining if the associated data item is unflagged, wherein:
(i) if the associated data item is unflagged, processing the real-time event and inserting information associated with such processing into the application-aware data stream; and
(ii) if the associated data item is not flagged, dropping the real-time event.
8. The method as described in claim 7 further including transferring the application-aware data stream from the first processing environment to a second processing environment.
9. The method as described in claim 8 wherein the second processing environment is remote from the first processing environment.
10. The method as described in claim 7 wherein a given data item in the set of data items is associated with data that is being modified by the application.
11. The method as described in claim 7 wherein the data structure is a sorted source tree.
12. The method as described in claim 7 wherein the application-aware data stream is a baseline that is generated when the data service is initiated for the application.
13. The method as described in claim 7 wherein the real-time event is a given failure event that occurs upon a given occurrence.
14. The method as described in claim 13 further including the step of taking a given action with respect to a given data item in the set of data items upon the given occurrence.
15. The method as described in claim 14 wherein the given action is a synchronization operation with respect to a second data store distinct from the local data store.