Multi-threaded transaction log for primary and restore/intelligence
View Patent ↗A unified system provides primary storage and in-line analytics-based data protection. Additional data intelligence and analytics gathered on protected data and prior analytics are stored in discovery points. The disclosed system implements multi-threaded log writes across primary and restore nodes with write gathering across file systems; nested directories such as may be used for storing virtual machine files, where every subdirectory has an associated file system for snapshot purposes; and cloning objects on demand with background metadata and data migration.
1. A method for processing in a data-intelligent storage system that includes a primary node storing primary data and in-line analytics, a restore node storing further analytics and intelligence, and a protection and analytics (PART) node, the method comprising:
receiving access requests from multiple concurrently executing threads at the PART node;
assigning a transaction identifier (ID) to each of the access requests;
logging the access requests at the PART node in a random access, multithreaded log; and
forwarding the access requests from the PART node to the primary node and restore node;
storing data associated with the access requests in a primary cache memory at the primary node;
storing data associated with the access requests in a restore cache memory at the restore node;
determining when a number of access requests in the random access, multithreaded log reaches a predetermined number;
issuing a synchronization command from the PART node to the primary and restore nodes;
in the primary node, flushing data from the primary cache memory to storage associated with a first file system in the primary node;
in the restore node, flushing data from the restore cache memory to storage associated with a second file system in the restore node; and
after flushing data in the primary and restore nodes, releasing entries in the random access, multithreaded log.
2. The method of claim 1 , further comprising:
logging access requests within a first single threaded sequential log at the primary node; and
logging access requests within a second single threaded sequential log at the restore node,
wherein the logging of access requests at the primary node and the logging of access requests at the restore node occur independently of the logging of access requests at the PART node.
3. The method of claim 1 , wherein the primary and restore cache memory in the primary and restore nodes, respectively, comprise volatile random access memory and wherein the storage associated with each of the first and second file systems comprises disk-based storage.
4. The method of claim 1 , additionally comprising:
prior to flushing data to at least one of the first and second file systems,
determining a failure condition is occurring in at least one of the first and second file systems; and
replaying the random access, multithreaded log in an order determined by the transaction IDs to bring at least one of the first and second file systems to a known consistent state.
5. The method of claim 1 , additionally comprising:
gathering write data at the PART node independently of the caching within the primary node and independently of the caching at the restore node.
6. A method for processing in a data-intelligent storage system that includes a primary node storing primary data and in-line analytics, a restore node storing further analytics and intelligence, and a protection and analytics (PART) node, the method comprising:
receiving access requests at the PART node from multiple concurrently executing threads;
assigning a transaction identifier (ID) to each of the access requests;
logging the access requests at the PART node in a random access, multithreaded log;
forwarding the access requests from the PART node to the primary node and restore node;
logging the access requests received at the primary node within a first single threaded sequential log;
logging the access requests received at the restore node within a second single threaded sequential log, wherein the logging of the access requests received at the primary node and the logging of the access requests at the restore node occur independently of the logging of access requests at the PART node;
storing data associated with the access requests in a primary cache at the primary node;
storing data associated with the access requests in a restore cache at the restore node;
determining when a number of access requests in the random access, multithreaded log in the PART node reaches a predetermined number;
issuing a synchronization command from the PART node to the primary and restore nodes;
at the primary node, flushing data from the primary cache to persistent storage associated with a first file system in the primary node;
in the restore node, flushing data from the restore cache to persistent storage associated with a second file system in the restore node; and
after flushing data in the primary and restore nodes, releasing entries in the random access, multithreaded log in the PART node.
7. The method of claim 6 , wherein the primary cache and the restore cache in the primary and restore nodes, respectively, comprise volatile random access memory and wherein the persistent storage associated with each of the primary and restore nodes comprises disk-based storage.
8. The method of claim 6 , additionally comprising:
prior to flushing data to at least one of the first and second file systems,
determining a failure condition is occurring in at least one of the first and second file systems; and
replaying the random access, multithreaded log in an order determined by the transaction IDs to bring at least one of the first and second file systems to a known consistent state.
9. A data-intelligent storage system, comprising:
a primary node for storing primary data and in-line analytics,
a restore node for storing further analytics and intelligence;
a protection and analytics (PART) node, the PART node for:
receiving access requests from multiple concurrently executing threads within the PART node;
assigning a transaction identifier (ID) to each of the access requests;
logging the access requests in a random access, multithreaded log; and
forwarding the access requests from the PART node to both the primary node and restore node;
the primary node further for logging the access requests within a first single threaded sequential log;
the restore node further for logging the access requests within a second single threaded sequential log, wherein logging the access requests by the primary node and restore node occur independently of the logging of the access requests by the PART node;
the primary node further comprises a primary cache memory for storing data associated with the access requests;
the restore node further comprises a restore cache memory for storing data associated with the access requests;
the PART node additionally for:
determining when a number of access requests in the random access, multithreaded log reaches a predetermined number; and
issuing a synchronization command to the primary and restore nodes;
the primary node additionally for flushing data from the primary cache to persistent storage associated with a first file system in the primary node;
the restore node additionally for flushing data from the restore cache to persistent storage associated with a second file system in the restore node; and
the PART node further for after flushing data in the primary and restore nodes, releasing entries in the random access, multithreaded log.
10. The system of claim 9 , wherein the PART node is additionally for gathering write data at a virtual file system layer.
11. The data-intelligent storage system of claim 9 , wherein the primary cache memory and the restore cache memory in the primary and restore nodes, respectively, comprise volatile random access memory and wherein the persistent storage associated with each of the first and second file systems comprises disk-based storage.
12. The data-intelligent storage system of claim 9 , wherein the PART node is additionally for:
prior to flushing data to at least one of the first and second file systems,
determining a failure condition is occurring in at least one of the first and second file systems; and
replaying the random access, multithreaded log in an order determined by the transaction IDs to bring at least one of the first and second file systems to a known consistent state.