Live restore for a data intelligent storage system
View Patent ↗A single system merges primary data storage, data protection, and intelligence. Intelligence is provided through in-line data analytics, and data intelligence and analytics are gathered on protected data and prior analytics, and stored in discovery points, all without impacting performance of primary storage. Real-time analysis is done in-line with the HA processing, enabling a variety of data analytics that are then used as part of a live restore operation. Data content can be live restored at an object or block level. Data recovery begins with metadata restoration, followed by near-instantaneous access to “hot” regions of data being restored, allowing site operation to continue or resume while a restore is ongoing.
1. A method for restoring data in a data-intelligent storage system that includes a primary storage node and intelligence storage node, each such node programmed to operate software, wherein
a. the primary node software:
i. intercepts data access requests;
ii. mirrors data to the intelligence node referenced in the data access requests to provide high availability for data included in the data access requests;
iii. executes in-line analytics on the data access requests to provide analytics metadata; and
iv. directs the data access requests to a primary storage pool;
b. the intelligence node software:
i. stores mirrored data to an intelligence storage pool;
ii. performs analytics on mirrored data and/or previously executed in-line analytics
iii. creates discovery points within the intelligence pool; and
c. upon indication of a need to restore data, the primary node and intelligence node cooperating to perform a live restore operation to the primary pool from the intelligence pool using the analytics metadata.
2. The method of claim 1 additionally comprising:
a. restoring only object metadata first as a foreground process;
b. allowing user I/O access to objects prior to user data associated with the object being completely restored; and
c. restoring user data as a background process.
3. The method of claim 2 additionally comprising:
prioritizing user data being restored based on the analytics metadata.
4. The method of claim 2 additionally comprising:
using the analytics metadata and user I/O operations to prioritize regions of a user data object being restored.
5. The method of claim 3 additionally comprising:
a. restoring multiple objects at a time; and
b. restoring user data using the analytics metadata and user I/O to determine a priority for restoring the multiple objects.
6. The method of claim 1 additionally comprising:
a. intercept subsequent live file requests; and
b. force live restore from a discovery point, wherein the discovery point further includes source data.
7. The method of claim 2 additionally comprising:
using a restore bitmap to track restore progress to allow random I/O to the object before the object is completely restored.
8. The method of claim 7 wherein the restore bitmap represents regions that support automatic atomic restoration.
9. The method of claim 1 additionally comprising:
a. obtaining a size of a file to be restored from the intelligence pool;
b. creating a new file in a RestorePath in the primary pool with an object identifier FIleOID;
d. restoring metadata for a user data object to be restored;
e. creating a RestoreLink file having an object identifier and a RestoreLinkOID;
f. saving at least the restore size, FileOID and zero restore bitmap in the RestoreLink file; and
g. saving the RestoreLinkOID on a corresponding attribute in new file being restored in the primary pool.
10. The method of claim 1 additionally comprising:
performing a live restore operation on an individual file.
11. The method of claim 10 wherein the file live restore operation further comprises:
a. restoring metadata related to the file atomically;
b. allowing user I/O to the file being restored;
c. continuing to restore user data related to the file as a background task.
12. The method of claim 10 wherein the file live restore operation further comprises:
a. prioritizing restore of individual blocks within the file being restored based on user I/O and the previously gathered analytics.
13. The method of claim 1 wherein the intelligence node software further performs a directory/file system live restore operation on a directory and/or file system level to restore multiple files and/or directories in parallel.
14. The method of claim 13 wherein the directory/file system live restore operation further comprises:
a. creating directory/file system metadata;
b. allowing user I/O to the file system/directory being live restored as a background task; and
c. restoring individual files as a file live restore operation.
15. The method of claim 13 wherein the directory/file system live restore operation further comprises:
a. prioritizing restoration of multiple files within a directory/file system based on user I/O and the previously gathered analytics.
16. The method of claim 1 additionally comprising perform block level live restore for restoring data extents from a recovery site to a primary site.
17. The method of claim 16 wherein the block level live restore further comprises:
a. restoring metadata for the site atomically, including system metadata for file systems and discovery points;
b. transferring an allocated extents map from a Virtual Disk Layer (VDL) on the recovery site to the VDL on the primary site; and
c. allowing user I/O to the primary site while performing restore of user data each extent as a background task.
18. The method of claim 17 additionally comprising:
a. pruning resulting metadata; and
b. resolving TRIM operations to the VDL as result of either such pruning or as a result of allowing user I/O to the site while it is being restored.
19. The method of claim 16 wherein the block level live restore further comprises:
a. prioritizing restore of each extent based on the user I/O and previously gathered analytics.