Consolidating analytics metadata
View Patent ↗A single system merges primary data storage, data protection, and intelligence. Intelligence is provided through in-line data analytics, and data intelligence and analytics are gathered on protected data and prior analytics, and stored in discovery points, all without impacting performance of primary storage. As data is written it is automatically mirrored for data protection as part of a High Availability (HA) process. Real-time analysis is done in-line with the HA processing, enabling a variety of data analytics. Data content can mined from within files or blocks. The gathered intelligence is used to tag objects with extended metadata, enabling both valuable search options and rapid restore options. Data recovery begins with metadata restoration, followed by near-instantaneous access to “hot” regions of data being restored, allowing site operation to continue or resume while a restore is ongoing.
1. A method for handling data in a data storage system comprising:
receiving a copy of data access requests and metadata concerning the data access requests;
executing further analytics on the metadata associated with the data access requests to provide further analytics data;
consolidating the metadata and the further analytics for storage as consolidated metadata and further analytics data in a storage device;
wherein the copy of data access requests is received from a primary node;
wherein an intelligence node provides the storage device for storing the consolidated metadata and further analytics data; and
wherein the intelligence node further
directs the data access requests to an intelligence pool; and
creates discovery points associating the consolidated metadata and further analytics data with an intelligence pool copy of the data.
2. The method of claim 1 wherein the primary node and intelligence node are co-located.
3. The method of claim 1 wherein the primary node and intelligence node are located remotely from one another.
4. The method of claim 1 wherein a single processing node implements the primary node and intelligence node, and the intelligence node executes when the single processing node is not occupied with functions of the primary node.
5. The method of claim 1 wherein the metadata includes one or more of:
extended information concerning the data access request;
who initiated a data access request;
how many modifications have occurred;
aggregate size of the modifications;
a data object name;
data object owner;
access control list; or
time of the data access request.
6. The method of claim 1 wherein the intelligence node further performs deeper analytics on the discovery points, including content extraction and/or analysis of information stored in the discovery points.
7. The method of claim 1 wherein the intelligence node further compares metadata related to two or more discovery points associated with data access requests occurring at two or more different times.
8. The method of claim 6 further comprising:
executing one or more rules on at least one of the change catalog, metadata, and extracted content information to perform an additional operation that include one or more of:
applying a rule that includes a filter to match content and an associated action;
tracking changes in the data or metadata over time;
applying sentiment attributes to an indexed document;
processing of a regular expression;
content transformation;
content analysis; or
triggering an action based on detected access or content wherein
the action further comprises one or more of:
data retention;
quarantine;
data extraction;
deletion;
data distribution;
alerts; or
other actions.
9. The method of claim 8 wherein a result of executing one or more rules provides complex analytics stored as additional metadata appended to one or more change catalogs.
10. A method for handling data in a data storage system comprising:
receiving a copy of data access requests and metadata concerning the data access requests;
executing further analytics on the metadata associated with the data access requests to provide further analytics data;
consolidating the metadata and the further analytics for storage as consolidated metadata and further analytics data in a storage device;
wherein the copy of data access requests is received from a primary node;
wherein an intelligence node provides the storage device for storing the consolidated metadata and further analytics data; and
wherein the intelligence node further provides:
selective restore of the primary node using the analytics metadata associated with a discovery point.
11. The method of claim 10 wherein the selective restore is in response to discovering a predetermined pattern in the data or metadata.
12. A method for handling data in a data storage system comprising:
receiving a copy of data access requests and metadata concerning the data access requests;
executing further analytics on the metadata associated with the data access requests to provide further analytics data; and
consolidating the metadata and the further analytics for storage as consolidated metadata and further analytics data in a storage device;
wherein the copy of data access requests is received from a primary node;
wherein an intelligence node provides the storage device for storing the consolidated metadata and further analytics data; and
wherein the intelligence node further controls replication of the data, associated analytics, and a linkage between the data and associated analytics to a remote location.
13. The method of claim 12 further comprising:
selective replication of analytics metadata at the remote location based on one or more of:
size of the analytics metadata;
complexity of a metadata production process;
priorities; or
processing load.
14. A primary storage, high availability and data analytics system comprising:
one or more processors programmed to operate intelligence node software, wherein the intelligence node software:
receives a copy of data access requests and extended metadata concerning the access requests;
directs the data access requests to physical storage media in an intelligence pool;
executes further analytics on the data associated with the data access request and extended metadata; and
consolidates the extended metadata and the further analytics for storage as consolidated metadata in the intelligence pool; and
wherein the intelligence node software further
creates discovery points associating the change catalog and the further analytics with an intelligence pool copy of the data.
15. The system of claim 14 wherein the copy of data access requests is received from a primary node.
16. The system of claim 15 wherein the primary node and intelligence node software are co-located.
17. The system of claim 15 wherein the primary node and intelligence node software are located remotely from one another.
18. The system of claim 15 wherein a single processing node implements the primary node and intelligence node, and the intelligence node executes during when the node is not occupied with functions of the primary node.
19. The system of claim 14 wherein the extended metadata includes one or more of:
extended information concerning the data access request;
who initiated a data access request;
how many modifications have occurred;
aggregate size of the modifications;
a data object name;
data object owner;
access control list; or
time of the data access request.
20. The system of claim 14 wherein the intelligence node software further performs deeper analytics on the discovery points, including content extraction and/or analysis of the data.
21. The system of claim 14 wherein the intelligence node software further compares metadata related to two or more discovery points associated with data access requests occurring at two or more different times.
22. The system of claim 14 wherein the intelligence node software further includes
a rule engine, for operating on at least one of the change catalog, extended metadata, and content information to perform an additional operation that includes one or more of:
applying a rule that includes a filter to match content and an associated action;
tracking changes in the data or metadata over time;
applying sentiment attributes to an indexed document;
processing of a regular expression;
content transformation;
content analysis; or
triggering an action based on detected access or content wherein the action further comprises one or more of:
data retention;
quarantine;
data extraction;
deletion;
data distribution;
alerts; or
other actions.
23. The system of claim 14 wherein the further analytics are provided by a rule engine and stored as additional metadata appended to one or more change catalogs.
24. The system of claim 14 wherein the intelligence node software further provides:
selective restore of the primary node using the analytics metadata associated with a discovery point.
25. The system of claim 24 wherein the selective restore is in response to discovering a predetermined pattern in the data or metadata.
26. The system of claim 14 wherein the intelligence node software further controls replication of the data, associated analytics, and a linkage between the data and associated analytics to a remote location.
27. The system of claim 26 wherein intelligence node software further:
selectively replicates analytics metadata at the remote location based on one or more of:
size of the analytics metadata;
complexity of a metadata production process;
priorities; or
processing load.