Offset segmentation for improved inline data deduplication
Systems and methods for processing data segments are disclosed. In one embodiment, such functionality includes buffering data received from a node (where the data is stored in a buffer as buffered data, an offset value is associated with the data, and a segment size is associated with the buffer), and determining whether the offset value is an integer multiple of the segment size. In response to determination that the offset value is an integer multiple of the segment size, processing the data in the buffer as a segment. Such functionality also includes determining whether the segment is a duplicate of data stored in a deduplicated data store and, in response to a determination that the segment is not a duplicate of data stored in the deduplicated data store, storing the segment in the deduplicated data store.
1. A method comprising:
buffering data received from a node, wherein
the data is stored in a buffer, as buffered data,
an offset value is associated with the data, and
a segment size is associated with the buffer;
determining whether the offset value is an integer multiple of the segment size;
in response to a determination that the offset value is an integer multiple of the segment size, processing the data in the buffer as a segment;
determining whether the segment is a duplicate of data stored in a deduplicated data store; and
in response to a determination that the segment is not a duplicate of data stored in the deduplicated data store, storing the segment in the deduplicated data store.
2. The method of claim 1 , further comprising:
determining whether the buffer is full, wherein
the determining whether the offset value is an integer multiple of the segment size is performed in response to a determination that the buffer is full.
3. The method of claim 1 , further comprising:
determining whether the buffer is full;
in response to a determination that the buffer is not full, determining if a trigger event has been received; and
in response to a determination that a trigger event has been received, deduplicating the buffered data by treating the buffered data as a segment without generating a modified segment.
4. The method of claim 1 , further comprising:
determining whether the buffer is full;
in response to a determination that the buffer is not full, determining if a trigger event has been received;
in response to a determination that a trigger event has not been received, receiving additional data from the node; and
buffering the additional data by storing the additional data in the buffer along with the buffered data, in order to create modified buffered data.
5. The method of claim 4 , further comprising:
subsequent to buffering the additional data, determining whether the buffer is full;
in response to a determination that the buffer is full, determining whether the offset value is an integer multiple of the segment size;
in response to determining that the offset value is not an integer multiple of the segment size, generating a modified segment;
determining whether the modified segment is a duplicate of data stored in a deduplicated data store; and
in response to a determination that the modified segment is not a duplicate of data stored in the deduplicated data store, storing the modified segment in the deduplicated data store.
6. The method of claim 5 , further comprising:
the generating the modified segment further comprises splitting the data into a first segment and a second segment, wherein
the first segment comprises a first number of units of the buffered data,
the first number of units is based on a modulo value,
the second segment comprises a second number of units of the buffered data, and
the second number of units is equivalent to a difference of the segment size and the modulo value;
moving the second segment to a subsequent buffer; and
after the moving, writing additional data to the subsequent buffer, wherein
the additional data is received from the node.
7. The method of claim 1 , further comprising
in response to the determination that the offset value is the integer multiple of the segment size, treating the buffered data as a segment without creating a modified segment.
8. A system comprising:
a microprocessor; and
a non-transient computer-readable storage medium, comprising computer instructions, executable by the microprocessor and configured to perform a method comprising
buffering data received from a node, wherein
the data is stored in a buffer, as buffered data,
an offset value is associated with the data, and
a segment size is associated with the buffer,
determining whether the offset value is an integer multiple of the segment size,
in response to a determination that the offset value is an integer multiple of the segment size, processing the data in the buffer as a segment,
determining whether the segment is a duplicate of data stored in a deduplicated data store, and
in response to a determination that the segment is not a duplicate of data stored in the deduplicated data store, storing the segment in the deduplicated data store.
9. The system of claim 8 , wherein the method further comprises:
determining whether the buffer is full, wherein
the determining whether the offset value is an integer multiple of the segment size is performed in response to a determination that the buffer is full.
10. The system of claim 8 , wherein the method further comprises:
determining whether the buffer is full;
in response to a determination that the buffer is not full, determining if a trigger event has been received; and
in response to a determination that a trigger event has been received, deduplicating the buffered data by treating the buffered data as a segment without generating a modified segment.
11. The system of claim 8 , wherein the method further comprises:
determining whether the buffer is full;
in response to a determination that the buffer is not full, determining if a trigger event has been received;
in response to a determination that a trigger event has not been received, receiving additional data from the node; and
buffering the additional data by storing the additional data in the buffer along with the buffered data, in order to create modified buffered data.
12. The system of claim 11 , wherein the method further comprises:
subsequent to buffering the additional data, determining whether the buffer is full;
in response to a determination that the buffer is full, determining whether the offset value is an integer multiple of the segment size;
in response to determining that the offset value is not an integer multiple of the segment size, generating a modified segment;
determining whether the modified segment is a duplicate of data stored in a deduplicated data store; and
in response to a determination that the modified segment is not a duplicate of data stored in the deduplicated data store, storing the modified segment in the deduplicated data store.
13. The system of claim 12 , wherein the method further comprises:
the generating the modified segment further comprises splitting the data into a first segment and a second segment, wherein
the first segment comprises a first number of units of the buffered data,
the first number of units is based on a modulo value,
the second segment comprises a second number of units of the buffered data, and
the second number of units is equivalent to a difference of the segment size and the modulo value;
moving the second segment to a subsequent buffer; and
after the moving, writing additional data to the subsequent buffer, wherein
the additional data is received from the node.
14. The system of claim 8 , wherein the method further comprises:
in response to the determination that the offset value is the integer multiple of the segment size, treating the buffered data as a segment without creating a modified segment.
15. A computer program product, comprising a plurality of instructions stored on a non-transient computer-readable storage medium, wherein the instructions are configured to execute a method comprising:
buffering data received from a node, wherein
the data is stored in a buffer, as buffered data,
an offset value is associated with the data, and
a segment size is associated with the buffer;
determining whether the offset value is an integer multiple of the segment size;
in response to a determination that the offset value is an integer multiple of the segment size, processing the data in the buffer as a segment;
determining whether the segment is a duplicate of data stored in a deduplicated data store; and
in response to a determination that the segment is not a duplicate of data stored in the deduplicated data store, storing the segment in the deduplicated data store.
16. The computer program product of claim 15 , wherein the method further comprises:
determining whether the buffer is full, wherein
the determining whether the offset value is an integer multiple of the segment size is performed in response to a determination that the buffer is full.
17. The computer program product of claim 15 , wherein the method further comprises:
determining whether the buffer is full;
in response to a determination that the buffer is not full, determining if a trigger event has been received; and
in response to a determination that a trigger event has been received, deduplicating the buffered data by treating the buffered data as a segment without generating a modified segment.
18. The computer program product of claim 15 , wherein the method further comprises:
determining whether the buffer is full;
in response to a determination that the buffer is not full, determining if a trigger event has been received;
in response to a determination that a trigger event has not been received, receiving additional data from the node; and
buffering the additional data by storing the additional data in the buffer along with the buffered data, in order to create modified buffered data.
19. The computer program product of claim 18 , wherein the method further comprises:
subsequent to buffering the additional data, determining whether the buffer is full;
in response to a determination that the buffer is full, determining whether the offset value is an integer multiple of the segment size;
in response to determining that the offset value is not an integer multiple of the segment size, generating a modified segment;
determining whether the modified segment is a duplicate of data stored in a deduplicated data store; and
in response to a determination that the modified segment is not a duplicate of data stored in the deduplicated data store, storing the modified segment in the deduplicated data store.
20. The computer program product of claim 19 , wherein the method further comprises:
the generating the modified segment further comprises splitting the data into a first segment and a second segment, wherein
the first segment comprises a first number of units of the buffered data,
the first number of units is based on a modulo value,
the second segment comprises a second number of units of the buffered data, and
the second number of units is equivalent to a difference of the segment size and the modulo value;
moving the second segment to a subsequent buffer; and
after the moving, writing additional data to the subsequent buffer, wherein
the additional data is received from the node.