Systems and methods for cloud-based data protection storage
A computer-implemented method for cloud-based data protection storage may include (1) identifying a request to maintain a data protection image for a volume of data, (2) identifying a third-party storage service to use for storing the data protection image, (3) receiving a stream of data to apply to the data protection image, (4) buffering a portion of the stream of data, (5) writing the buffered portion of the stream to the third-party storage service as a single storage object, and (6) associating the single storage object with the buffered portion of the stream in a data map. Various other methods, systems, and computer-readable media are also disclosed.
1. A computer-implemented method for cloud-based data protection storage, at least a portion of the method being performed by a computing device comprising at least one processor, the method comprising:
identifying a data protection image for a volume of data stored using a third-party storage service;
receiving a stream of data from at least one data mover to apply to the data protection image;
buffering a portion of the stream of data received from the data mover;
determining that the buffered portion of the stream of data received from the data mover overwrites a corresponding portion of the data protection image that corresponds to a portion of a previous storage object stored on the third-party storage service such that the corresponding portion of the previous storage object is no longer valid;
writing the buffered portion of the stream of data received from the data mover to the third-party storage service as a single storage object;
identifying a data map entry that maps an offset within the previous storage object stored on the third-party storage service to an offset within a previous stream of data corresponding to the previous storage object, thereby enabling reconstruction of a portion of the previous stream of data represented by the previous storage object in a correct sequence within the previous stream;
updating, based on the determination that the buffered portion of the stream of data overwrites the corresponding portion of the data protection image, the offset within the previous storage object, as recorded in the data map entry to reflect the overwriting by the buffered portion of the stream of data.
2. The computer-implemented method of claim 1 , further comprising optimizing a data protection system for the third-party storage service by manipulating a size of a buffer for the stream of data received from the data mover, wherein:
receiving the stream of data from the data mover comprises receiving the stream of data as part of an operation of the data protection system;
buffering the portion of the stream of data received from the data mover comprises buffering the stream of data received from the data mover until the buffer is full.
3. The computer-implemented method of claim 2 , wherein the size of the buffer is configured based on the third-party storage service.
4. The computer-implemented method of claim 1 , wherein buffering the portion of the stream of data received from the data mover comprises buffering the stream of data received from the data mover until the stream of data received from the data mover provides data out of sequence.
5. The computer-implemented method of claim 4 , wherein buffering the stream of data until the stream of data provides data out of sequence comprises buffering the stream of data until the stream of data includes an unbroken sequential write operation followed by a write operation for a different location on the volume of data.
6. The computer-implemented method of claim 5 , further comprising delimiting the unbroken sequential write operation within the buffer.
7. The computer-implemented method of claim 1 , further comprising:
determining that the corresponding portion of the previous storage object no longer comprises valid data;
determining that a remaining portion of the previous storage object still comprises valid data;
leaving the previous storage object on the third-party storage service based on the determination that the remaining portion of the previous storage object still comprises valid data;
updating, based on the determination that the remaining portion of the previous storage object still comprises valid data, the data map entry to specify the offset within the previous storage object at which valid data begins and a length of the valid data within the previous storage object such that the data map entry reflects that only a portion of the previous storage object is valid.
8. The computer-implemented method of claim 7 , further comprising:
identifying a request to retrieve the data protection image;
analyzing a data map comprising the data map entry to identify storage objects on the third-party storage service that are needed to retrieve the data protection image;
determining that the storage objects on the third-party storage service that are needed to retrieve the data protection image comprise the previous storage object and the single storage object;
reconstructing the data protection image with the remaining portion of the previous storage object and the single storage object using the data map entry.
9. The computer-implemented method of claim 1 , wherein the third-party storage service does not allow random access writes.
10. The computer-implemented method of claim 1 , wherein the stream of data comprises a stream of data updates to the volume of data.
11. The computer-implemented method of claim 1 , wherein the previous storage object was created by:
receiving the previous stream of data to apply to the data protection image;
buffering a portion of the previous stream of data;
writing the buffered portion of the previous stream of data to the third-party storage service as a single storage object, the single storage object comprising the previous storage object.
12. A system for cloud-based data protection storage, the system comprising:
an identification module programmed to identify a data protection image for a volume of data stored using a third-party storage service;
a buffering module programmed to:
receive a stream of data from at least one data mover to apply to the data protection image;
buffer a portion of the stream of data received from the data mover;
a storage module programmed to:
determine that the buffered portion of the stream of data received from the data mover overwrites a corresponding portion of the data protection image that corresponds to a portion of a previous storage object on the third-party storage service such that the corresponding portion of the previous storage object is no longer valid;
write the buffered portion of the stream of data received from the data mover to the third-party storage service as a single storage object;
identify a data map entry that maps an offset within the previous storage object stored on the third-party storage service to an offset within a previous stream of data corresponding to the previous storage object, thereby enabling reconstruction of a portion of the previous stream of data represented by the previous storage object in a correct sequence within the previous stream;
update, based on the determination that the buffered portion of the stream of data overwrites the corresponding portion of the data protection image, the offset within the previous storage object, as recorded in the data map entry to reflect the overwriting by the buffered portion of the stream of data;
at least one processor configured to execute the identification module, the buffering module, and the storage module.
13. The system of claim 12 , wherein:
the buffering module is further programmed to optimize a data protection system for the third-party storage service by manipulating a size of a buffer for the stream of data received from the data mover;
the buffering module is programmed to receive the stream of data from the data mover by receiving the stream of data as part of an operation of the data protection system;
the buffering module is programmed to buffer the portion of the stream of data received from the data mover by buffering the stream of data received from the data mover until the buffer is full.
14. The system of claim 13 , wherein the size of the buffer is configured based on the third-party storage service.
15. The system of claim 12 , wherein the buffering module is programmed to buffer the portion of the stream of data received from the data mover by buffering the stream of data received from the data mover until the stream of data received from the data mover provides data out of sequence.
16. The system of claim 12 , wherein the storage module is further programmed to:
determine that the corresponding portion of the previous storage object no longer comprises valid data;
determine that a remaining portion of the previous storage object still comprises valid data;
leave the previous storage object on the third-party storage service based on the determination that the remaining portion of the previous storage object still comprises valid data;
update, based on the determination that the remaining portion of the previous storage object still comprises valid data, the data map entry to specify the offset within the previous storage object at which valid data begins and a length of the valid data within the previous storage object such that the data map entry reflects that only a portion of the previous storage object is valid.
17. The system of claim 16 , wherein the storage module is further programmed to:
identify a request to retrieve the data protection image;
analyze a data map comprising the data map entry to identify storage objects on the third-party storage service that are needed to retrieve the data protection image;
determine that the storage objects on the third-party storage service that are needed to retrieve the data protection image comprise the previous storage object and the single storage object;
reconstruct the data protection image with the remaining portion of the previous storage object and the single storage object using the data map entry.
18. The system of claim 12 , wherein the third-party storage service does not allow random access writes.
19. The system of claim 12 , wherein the stream of data comprises a stream of data updates to the volume of data.
20. A non-transitory computer-readable-storage medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to:
identify a data protection image for a volume of data stored using a third-party storage service;
receive a stream of data from at least one data mover to apply to the data protection image;
buffer a portion of the stream of data received from the data mover;
determine that the buffered portion of the stream of data received from the data mover overwrites a corresponding portion of the data protection image that corresponds to a portion of a previous storage object stored on the third-party storage service such that the corresponding portion of the previous storage object is no longer valid;
write the buffered portion of the stream of data received from the data mover to the third-party storage service as a single storage object;
identify a data map entry that maps an offset within the previous storage object stored on the third-party storage service to an offset within a previous stream of data corresponding to the previous storage object, thereby enabling reconstruction of a portion of the previous stream of data represented by the previous storage object in a correct sequence within the stream;
update, based on the determination that the buffered portion of the stream of data overwrites the corresponding portion of the data protection image, the offset within the previous storage object, as recorded in the data map entry to reflect the overwriting by the buffered portion of the stream of data.