Performance of deduplication storage systems
Disclosed herein are methods, systems, and processes to improve backup and restore performance in deduplication storage environments. A metadata stream that includes data segment offsets that are associated with data segments of a previous backup image and indicate data segment boundaries is received. An offset for an include operation is determined. The include operation references one or more data segments, and is part of a request to perform a backup operation. The backup operation is performed by modifying the include operation, if the offset involves one or more partial data segments.
1. A system comprising:
one or more processors; and
a memory coupled to the one or more processors, wherein the memory stores program instructions executable by the one or more processors to
perform a backup operation, comprising further program instructions executable by the one or more processors to
send a request for an existing metadata stream, wherein
the existing metadata stream comprises a plurality of data segment offsets associated with a plurality of data segments of a previous backup image,
the previous backup image was created prior to the backup operation, and
the plurality of data segment offsets indicate a plurality of data segment boundaries,
receive the existing metadata stream,
determine an offset for an include operation, wherein
the offset is one of a start offset or an end offset,
the include operation references one or more data segments of the plurality of data segments,
the include operation is part of a request to perform the backup operation,
at least one data segment is to be included in the backup operation as a result of the include operation, and
the at least one data segment was not in the previous backup image at the offset,
determine whether the include operation will result in a data segment of the one or more data segments becoming one or more partial data segments, wherein
the determining is based, at least in part, on the existing metadata stream and the offset, and
in response to a determination that the include operation will result in the data segment of the one or more data segments becoming the one or more partial data segments, create a modified include operation, wherein
the further program instructions executable by the one or more processors to create comprise program instructions executable by the one or more processors to
modify the include operation to include the one or more partial data segments with the at least one data segment,
determine whether the data segment is a first data segment or a last data segment of the one or more data segments,
in response to a determination that the data segment is a first data segment,
align the offset of the include operation with a data segment boundary of the plurality of data segment boundaries, such that a data segment boundary of a partial data segment of the one or more partial data segments are aligned on a first data segment boundary of the plurality of data segment boundaries, or
in response to a determination that the data segment is a first data segment,
align the offset of the include operation with the data segment boundary of the plurality of data segment boundaries, such that the data segment boundary of a partial data segment of the one or more partial data segments are aligned on a last data segment boundary of the plurality of data segment boundaries.
2. The system of claim 1 , wherein
the existing metadata stream represents the previous backup image in its entirety, and
the one or more partial data segments are aligned by
aligning the offset of the include operation with a data segment boundary of the plurality of data segment boundaries.
3. The system of claim 2 , wherein the further program instructions executable by the one or more processors to align comprise program instructions executable to:
move the start offset forward, if the first data segment of the one or more data segments comprises the partial data segment, or
move the end offset backward, if the last data segment of the one or more data segments comprises the partial data segment.
4. The system of claim 3 , further comprising program instructions executable by the one or more processors to:
generate a modified backup stream by performing the modified include operation as part of the backup operation.
5. The system of claim 4 , further comprising program instructions executable by the one or more processors to:
transmit the partial data segment instead of the first data segment to a media server, as part of the modified backup stream, if the first data segment of the one or more data segments comprises the partial data segment; and
transmit the partial data segment instead of the last data segment to a media server, as part of the modified backup stream, if the last data segment of the one or more data segments comprises the partial data segment.
6. A method comprising:
performing a backup operation, comprising
sending a request for an existing metadata stream, wherein
the existing metadata stream comprises a plurality of data segment offsets associated with a plurality of data segments of a previous backup image,
the previous backup image was created prior to the backup operation, and
the plurality of data segment offsets indicate a plurality of data segment boundaries,
receiving the existing metadata stream,
determining an offset for an include operation, wherein
the offset is one of a start offset or an end offset,
the include operation references one or more data segments of the plurality of data segments,
the include operation is part of a request to perform the backup operation,
at least one data segment is to be included in the backup operation as a result of the include operation, and
the at least one data segment was not in the previous backup image at the offset,
determining whether the include operation will result in a data segment of the one or more data segments becoming one or more partial data segments, wherein
the determining is based, at least in part, on the existing metadata stream and the offset, and
in response to a determination that the include operation will result in the data segment of the one or more data segments becoming the one or more partial data segments, creating a modified include operation, wherein the creating comprises
modifying the include operation to include the one or more partial data segments with the at least one data segment,
determining whether the data segment is a first data segment or a last data segment of the one or more data segments,
in response to a determination that the data segment is a first data segment,
the modifying aligns the offset of the include operation with a data segment boundary of the plurality of data segment boundaries, such that a data segment boundary of a partial data segment of the one or more partial data segments are aligned on a first data segment boundary of the plurality of data segment boundaries, or
in response to a determination that the data segment is a last data segment,
the modifying aligns the offset of the include operation with the data segment boundary of the plurality of data segment boundaries, such that the data segment boundary of a partial data segment of the one or more partial data segments are aligned on a last data segment boundary of the plurality of data segment boundaries.
7. The method of claim 6 , wherein
the existing metadata stream represents the previous backup image in its entirety, and
the one or more partial data segments are aligned by
aligning the offset of the include operation with a data segment boundary of the plurality of data segment boundaries.
8. The method of claim 7 , wherein the aligning comprises:
moving the start offset forward, if the first data segment of the one or more data segments comprises the partial data segment.
9. The method of claim 8 , wherein the aligning further comprises:
moving the end offset backward, if the last data segment of the one or more data segments comprises the partial data segment.
10. The method of claim 9 , further comprising:
generating a modified backup stream by performing the modified include operation as part of the backup operation.
11. The method of claim 10 , further comprising:
transmitting the partial data segment instead of the first data segment to a media server, as part of the modified backup stream, if the first data segment of the one or more data segments comprises the partial data segment.
12. The method of claim 10 , further comprising:
transmitting the partial data segment instead of the last data segment to a media server, as part of the modified backup stream, if the last data segment of the one or more data segments comprises the partial data segment.
13. A non-transitory computer readable storage medium comprising program instructions executable to:
perform a backup operation, comprising further program instructions executable to send a request for an existing metadata stream, wherein
the existing metadata stream comprises a plurality of data segment offsets associated with a plurality of data segments of a previous backup image,
the previous backup image was created prior to the backup operation, and
the plurality of data segment offsets indicate a plurality of data segment boundaries,
receive the existing metadata stream,
determine an offset for an include operation, wherein
the offset is one of a start offset or an end offset,
the include operation references one or more data segments of the plurality of data segments,
the include operation is part of a request to perform the backup operation,
at least one data segment is to be included in the backup operation as a result of the include operation, and
the at least one data segment was not in the previous backup image at the offset,
determine whether the include operation will result in a data segment of the one or more data segments becoming one or more partial data segments, wherein
the determining is based, at least in part, on the existing metadata stream and the offset, and
in response to a determination that the include operation will result in the data segment of the one or more data segments becoming the one or more partial data segments, create a modified include operation, wherein
the further program instructions executable to create comprise program instructions executable to
modify the include operation to include the one or more partial data segments with the at least one data segment,
determine whether the data segment is a first data segment or a last data segment of the one or more data segments,
in response to a determination that the data segment is a first data segment,
align the offset of the include operation with a data segment boundary of the plurality of data segment boundaries, such that a data segment boundary of a partial data segment of the one or more partial data segments are aligned on a first data segment boundary of the plurality of data segment boundaries, or
in response to a determination that the data segment is a first data segment,
align the offset of the include operation with the data segment boundary of the plurality of data segment boundaries, such that the data segment boundary of a partial data segment of the one or more partial data segments are aligned on a last data segment boundary of the plurality of data segment boundaries.
14. The non-transitory computer readable storage medium of claim 13 , wherein
the existing metadata stream represents the previous backup image in its entirety, and
the one or more partial data segments are aligned by
aligning the offset of the include operation with a data segment boundary of the plurality of data segment boundaries.
15. The non-transitory computer readable storage medium of claim 14 , wherein the further program instructions executable to align comprise program instructions executable to:
move the start offset forward, if the first data segment of the one or more data segments comprises the partial data segment, or
move the end offset backward, if the last data segment of the one or more data segments comprises the partial data segment.
16. The non-transitory computer readable storage medium of claim 15 , further comprising program instructions executable to:
generate a modified backup stream by performing the modified include operation as part of the backup operation.
17. The non-transitory computer readable storage medium of claim 16 , further comprising program instructions executable to:
transmit the partial data segment instead of the first data segment to a media server, as part of the modified backup stream, if the first data segment of the one or more data segments comprises the partial data segment; and
transmit the partial data segment instead of the last data segment to a media server, as part of the modified backup stream, if the last data segment of the one or more data segments comprises the partial data segment.