Processing I/O operations in parallel while maintaining read/write consistency using range and priority queues in a data protection system
In one example, a method includes receiving a stream that identifies I/O operations requested to be performed with respect to a data grouping, and identifying, in the stream, random I/O requests and sequential I/O requests. Next, the stream is split into an incoming queue containing the sequential I/O requests and a priority/processing queue containing the random I/O requests. A metadata load queue is then defined by assigning each of the random I/O requests to a respective range of metadata. The metadata of each range is loaded, and the random I/O requests re-queued. Using the loaded metadata, a random I/O operation is performed that corresponds to one of the random I/O requests. Finally, the random I/O operation and a sequential I/O operation are performed concurrently with each other.
1. A method, comprising:
receiving a stream that identifies I/O operations requested to be performed with respect to a data grouping;
identifying, in the stream, random I/O requests and sequential I/O requests;
queuing the sequential I/O requests in a priority/processing queue and processing the sequential I/O requests immediately without waiting for loading of any metadata;
defining a metadata load queue by assigning each of the random I/O requests to a respective range of metadata;
loading the metadata of each range;
re-queuing the random I/O requests;
using the loaded metadata, performing a random I/O operation that corresponds to one of the random I/O requests; and
performing the random I/O operation and a sequential I/O operation concurrently with each other.
2. The method as recited in claim 1 , wherein each metadata range corresponds to a respective set of one or more data segments.
3. The method as recited in claim 1 , wherein the metadata and the data segments are stored in different respective types of storage.
4. The method as recited in claim 1 , wherein the random I/O requests are re-queued in the priority/processing queue in the same order, relative to each other, in which they were initially received as part of the stream.
5. The method as recited in claim 1 , wherein metadata included in the metadata ranges comprises part of a metadata tree that represents the data grouping.
6. The method as recited in claim 1 , further comprising creating a metadata tree that represents the data grouping, the metadata tree including data segments of the data grouping and also including metadata embraced by the metadata ranges.
7. The method as recited in claim 1 , further comprising creating a unique reference for each data segment in the data grouping, wherein the unique references collectively comprise the metadata.
8. The method as recited in claim 1 , wherein the requested I/O operations comprise a backup operation and/or a restore operation.
9. The method as recited in claim 1 , wherein the requested I/O operations include a backup operation, and data deduplication is performed as part of that backup operation.
10. The method as recited in claim 1 , further comprising caching the metadata to SSD.
11. A non-transitory storage medium having stored therein computer-executable instructions which, when executed by one or more hardware processors, perform the following:
receiving a stream that identifies I/O operations requested to be performed with respect to a data grouping;
identifying, in the stream, random I/O requests and sequential I/O requests;
queuing the sequential I/O requests in a priority/processing queue and processing the sequential I/O requests immediately without waiting for loading of any metadata;
defining a metadata load queue by assigning each of the random I/O requests to a respective range of metadata;
loading the metadata of each range;
re-queuing the random I/O requests;
using the loaded metadata, performing a random I/O operation that corresponds to one of the random I/O requests; and
performing the random I/O operation and a sequential I/O operation concurrently with each other.
12. The non-transitory storage medium as recited in claim 11 , wherein each metadata range corresponds to a respective set of one or more data segments.
13. The non-transitory storage medium as recited in claim 11 , wherein the metadata and the data segments are stored in different respective types of storage.
14. The non-transitory storage medium as recited in claim 11 , wherein the random I/O requests are re-queued in the priority/processing queue in the same order, relative to each other, in which they were initially received as part of the stream.
15. The non-transitory storage medium as recited in claim 11 , wherein metadata included in the metadata ranges comprises part of a metadata tree that represents the data grouping.
16. The non-transitory storage medium as recited in claim 11 , wherein the processes further comprise creating a metadata tree that represents the data grouping, the metadata tree including data segments of the data grouping and also including metadata embraced by the metadata ranges.
17. The non-transitory storage medium as recited in claim 11 , wherein the processes further comprise creating a unique reference for each data segment in the data grouping, wherein the unique references collectively comprise the metadata.
18. The non-transitory storage medium as recited in claim 11 , wherein the requested I/O operations include a backup operation, and data deduplication is performed as part of that backup operation.
19. The non-transitory storage medium as recited in claim 11 , wherein the processes further comprise caching the metadata to SSD.
20. A physical device, wherein the physical device comprises:
one or more hardware processors; and
the non-transitory storage medium as recited in claim 11 .