Word aware content defined chunking
One example method includes, in a data buffer that includes one or more words and whitespaces, calculating a hash value of data in a window that is movable within the data buffer, comparing the hash value to a mask, and when the hash value matches the mask, identifying a position of the window in the data buffer as a chunk anchor position, searching for a whitespace nearest the chunk anchor position, and designating an offset of the whitespace as a segment boundary.
1. A method, comprising:
moving a window from a first position in a data buffer to a second position in the data buffer, and the data buffer includes one or more words;
calculating a hash value of data in the window when the window is in the second position;
checking a byte that has entered the window, as a result of a movement of the window from the first position to the second position, to determine whether the byte is whitespace; and
when the hash value is a greatest hash value seen up to a current position of the window, and when the byte is determined to be whitespace, setting a candidate offset to a whitespace offset, and the candidate offset denotes a possible segment boundary that does not fall within any word in the data buffer.
2. The method as recited in claim 1 , wherein when the hash value is not the greatest hash value seen up to the position of the window, and the byte is determined not to be whitespace, setting a candidate offset to a window offset.
3. The method as recited in claim 1 , wherein when the hash value is the greatest hash value seen up to the position of the window, and the byte is determined not to be whitespace, setting a candidate offset to a window offset.
4. The method as recited in claim 1 , wherein when the candidate offset is not set to a whitespace offset, identifying a closest whitespace to the candidate offset and designating the closest whitespace as a segment boundary.
5. The method as recited in claim 1 , wherein the window movement is either right to left, or left to right, in the data buffer.
6. The method as recited in claim 1 , wherein when designation of the whitespace offset as a segment boundary violates a maximum or minimum segment size, searching for an alternative whitespace as a segment boundary.
7. A method, comprising:
in a data buffer that includes one or more words and whitespaces, calculating a hash value of data in a window that is movable within the data buffer;
comparing the hash value to a mask, and when the hash value matches the mask, identifying a position of the window in the data buffer as a chunk anchor position;
searching for a whitespace nearest the chunk anchor position; and
designating an offset of the whitespace as a segment boundary to ensure that the segment boundary does not fall within any of the words in the data buffer.
8. The method as recited in claim 7 , wherein the searching comprises searching the data buffer by traversing a portion of the data buffer to locate the whitespace.
9. The method as recited in claim 7 , wherein when designation of the whitespace offset as a segment boundary violates a maximum or minimum segment size, searching for an alternative whitespace as a segment boundary.
10. The method as recited in claim 7 , wherein the chunk anchor position falls within one of the words of the data buffer.
11. The method as recited in claim 7 , wherein the data buffer is bounded by a minimum segment size and a maximum segment size.
12. The method as recited in claim 7 , wherein movement of the window within the data buffer is a bytewise movement.
13. The method as recited in claim 7 , wherein the segment boundary is a beginning of a segment, or an end of a segment.
14. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
in a data buffer that includes one or more words and whitespaces, calculating a hash value of data in a window that is movable within the data buffer;
comparing the hash value to a mask, and when the hash value matches the mask, identifying a position of the window in the data buffer as a chunk anchor position;
searching for a whitespace nearest the chunk anchor position; and
designating an offset of the whitespace as a segment boundary to ensure that the segment boundary does not fall within any of the words in the data buffer.
15. A non-transitory storage medium as recited in claim 14 , wherein the searching comprises searching the data buffer by traversing a portion of the data buffer to locate the whitespace.
16. A non-transitory storage medium as recited in claim 14 , wherein when designation of the whitespace offset as a segment boundary violates a maximum or minimum segment size, searching for an alternative whitespace as a segment boundary.
17. A non-transitory storage medium as recited in claim 14 , wherein the chunk anchor position falls within one of the words of the data buffer.
18. A non-transitory storage medium as recited in claim 14 , wherein the data buffer is bounded by a minimum segment size and a maximum segment size.
19. A non-transitory storage medium as recited in claim 14 , wherein movement of the window within the data buffer is bytewise movement.
20. A non-transitory storage medium as recited in claim 14 , wherein the segment boundary is a beginning of a segment, or an end of a segment.