Method and system for real-time event journaling to provide enterprise data services
View Patent ↗A data management system or “DMS” provides a wide range of data services to data sources associated with a set of application host servers. The data management system typically comprises one or more regions, with each region having one or more clusters. A given cluster has one or more nodes that share storage. To facilitate the data service, a host driver embedded in an application server connects an application and its data to a cluster. The host driver provides a method and apparatus for capturing real-time data modifications and application state notifications and, in response, generating data transactions in the form of an event journal that is provided to the data management system. The driver functions to translate traditional file/database/block I/O into a continuous, application-aware, output data stream. Using the streams generated in this manner, the DMS offers a wide range of data services that include, by way of example only: data protection (and recovery), and disaster recovery (data distribution and data replication).
1. Apparatus for use in providing a data service, comprising:
a filter, located between an application and an associated data storage;
a module, connectable to the filter, that identifies to the filter a set of one or more requests or responses expected to occur as the application interfaces to the data storage, and a given action to take upon occurrence of each request or response, wherein a given request or response has associated therewith one or more data changes;
a queue connectable to the filter to receive and store one or more data events generated as a function of the given action, wherein a given data event has associated therewith one or more of the data changes, together with metadata; and
a processor connectable to the queue to receive the data events, and to process and consolidate the data events by a concatenation operation into a continuous, application-aware, data stream comprising a sequence of application consistent checkpoints, wherein a given application consistent checkpoint is generated as a result of a consolidation of data events and has associated therewith information identifying the data events, the metadata, and the one or more data changes;
wherein, during processing of a given data event, the processor determines if a particular data event signifies an application consistent checkpoint and, if not, the processor holds the particular data event until the particular data event is subsequently associated with the given application consistent checkpoint.
2. The apparatus as described in claim 1 further including an application module, connectable to the application, and that identifies an application level event, and a given action to take upon occurrence of the application level event.
3. The apparatus as described in claim 1 wherein the processor is further operative to generate a given event and to store the given event in the queue for subsequent processing.
4. The apparatus as described in claim 3 wherein the given event is an internal event that must be processed into an outbound event before being included in the data stream.
5. The apparatus as described in claim 1 wherein the data events include at least one file system event.
6. The apparatus as described in claim 1 wherein the data events include at least one block device I/O event.
7. The apparatus as described in claim 1 wherein the given action retrieves given data from the data storage and forwards the given data to the queue.
8. The apparatus as described in claim 1 wherein the continuous, application-aware, data stream comprises one or more of the following data types: file or block data, byte-level deltas, and transaction data.
9. The method as described in claim 1 wherein the metadata is one of: user information, document properties, file type, and date and time information.
10. Apparatus for use in providing a data service, comprising:
a filter, located between an application and an associated data storage;
an input/output module, connectable to the filter, that identifies to the filter a set of one or more requests or responses expected to occur as the application interfaces to the data storage, and a given action to take upon occurrence of each request or response, wherein a given request or response has associated therewith one or more data changes;
a database module, connectable to a database associated with an application that stores data in the database;
a queue connectable to the filter and the database module to receive and store one or more data events generated by the database module and the filter as a function of the given action, wherein a given data event has associated therewith one or more of the data changes, together with metadata; and
a processor connectable to the queue to receive the data events, and to process and consolidate the data events by a concatenation operation into a continuous, application-aware, data stream comprising a sequence of application consistent checkpoints, wherein a given application consistent checkpoint is generated as a result of a consolidation of data events and has associated therewith information identifying the data events, the metadata, and the one or more data changes;
wherein the processing includes determining if a particular data event signifies an application consistent checkpoint and, if not, hold the particular data event from consolidation until the particular data event is subsequently associated with the given application consistent checkpoint.
11. The apparatus as described in claim 10 wherein the processor is further operative to generate a given event and to store the given event in the queue for subsequent processing.
12. The apparatus as described in claim 11 wherein the given event is an internal event that must be processed into an outbound event before being included in the data stream.
13. The apparatus as described in claim 10 wherein the data events include at least one database event.
14. The apparatus as described in claim 10 wherein the continuous, application-aware, data stream comprises one or more of the following data types: file or block data, byte-level deltas, and transaction data.
15. A method of monitoring an application in communication with a data storage to facilitate provision of a data service, comprising:
identifying a set of one or more requests or responses expected to occur as the application interfaces to the data storage, and a given action to take upon occurrence of each request or response, wherein a given request or response has associated therewith one or more data changes;
as the application interfaces to the data storage, receiving and storing one or more data events generated as a function of a given action, wherein a given data event has associated therewith one or more of the data changes, together with metadata; and
processing and consolidating the data events by a concatenation operation into a continuous, application-aware, data stream comprising a sequence of one or more application consistent checkpoints, wherein a given application consistent checkpoint is generated as a result of a consolidation of data events and has associated therewith information identifying the data events, the metadata, and the one or more data changes;
wherein the processing includes determining if a particular data event signifies an application consistent checkpoint and, if not, holding the particular data event from consolidation until the particular data event is subsequently associated with the given application consistent checkpoint.
16. The method as described in claim 15 wherein the data service is data protection.
17. The method as described in claim 15 wherein the data service is data replication.
18. The method as described in claim 15 wherein the data service is data distribution.
19. The method as described in claim 15 wherein the metadata is one of: user information, document properties, file type, and date and time information.